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Abstract 

Air  traffic  controllers  are  often  required  to  simultaneously  communicate  with 
several  aircraft  over  multiple  radio  frequencies.  As  a  result,  during  peak  loading,  it  is 
common  for  the  controller  to  receive  multiple  concurrent  communications,  each  from  a 
different  aircraft,  making  it  difficult  to  discern  audio  messages  received  from  multiple 
pilots  simultaneously.  To  address  this  problem,  a  modified  air  traffic  control  (ATC) 
interface  was  prototyped  with  the  goal  of  increasing  controller-to-pilot  communication 
efficiency.  This  prototype  included  supplementary  text  cueing  which  was  provided  by  a 
hypothetical  automated  text  to  speech  system  in  an  on-screen  text  box  for  controller 
reference  in  the  event  of  an  obscured  or  indiscernible  radio  call.  The  prototype  was  then 
evaluated  by  a  group  of  35  participants,  all  with  ATC  experience  including  12  students 
and  23  instructors  at  the  Air  Force  controllers’  school  house  Keesler  AFB,  MS.  The  text 
cueing  improved  controllers’  comprehension  of  pilots’  transmissions. 
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SUPPLEMENTARY  COMPUTER  GENERATED  CUEING  TO  ENHANCE 

CONTROLLER  EFFICIENCY 


I.  Introduction 


General  Issue 

Use  of  technology  is  increasing  every  day.  Advances  in  hardware  and  software 
offer  promising  opportunities  for  automating  or  augmenting  a  greater  range  of 
information  gathering,  information  processing,  decision  making,  and  control  functions 
than  has  been  possible  in  the  past.  Along  with  these  advances  comes  the  question  of  the 
degree  to  which  emerging  hardware  and  software  systems  impact  human  perfonnance.  It 
is  the  influence  of  the  technology  on  user's  efficiency  during  a  decision  making  process 
that  is  interesting  to  this  researcher,  specifically  in  the  air  traffic  control  environment 
where  the  effectiveness  of  the  tool  facilitates  critical  decisions. 

Many  ways  to  automate  routine  tasks  and  to  implement  automated  aids  have  been 
investigated  to  optimize  the  controller's  workload  (Erzberger,  2001;  Oprins  et  ah,  2009). 
Much  of  the  effort  to  modernize  air-traffic  control  services  is  focused  on  the  development 
and  implementation  of  effective  computer-based  decision  support  tools  (Weber  et  al, 
2007;  Couluris,  Schleicher,  and  Weidner,  1998).  A  common  challenge  in  the  application 
of  decision  support  tools  for  air  traffic  control  is  to  assure  an  adequate  human  factors 
design  and  also  improve  the  time-critical  aspect  of  controller  decision  making. 

This  research  has  implications  for  development  of  systems  and  tools  for  the 
increasingly  congested  environment  of  air  traffic  control,  especially  when  it  comes  to 
controllers.  Air  traffic  controllers  play  a  critical  role  on  Air  Traffic  Control  System 
(ATCS).  The  human  aspects  of  developing  decision  aids  to  enhance  Air  Traffic  Control 
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(ATC)  efficiency  need  to  be  carefully  considered  and  carried  out  through  detailed  study 
about  human  factors  and  how  they  could  be  impacted  by  the  tool  to  continually  ensure 
safety  of  aircrews  and  passengers  within  the  ATCS. 


Problem  Statement 

“The  safe,  orderly,  and  expeditious  flow  of  air  traffic”  (Hopkin,  1991a)  have  long 
been  the  fundamental  objectives  of  ATC  (see  Federal  Aviation  Administration,  1989a). 
Controllers  have  a  critical  role  in  ensuring  the  achievement  of  these  traditional 
fundamentals.  The  complex  and  dynamic  environment  that  controllers  deal  with  requires 
that  they  continually  revise  all  the  information  available  to  base  and  make  their  decisions. 
The  need  of  accurate  information  availability,  decision  making  and  action  execution  in 
time-critical  ATC  is  explained  well  in  a  2004  issue  of  The  International  Journal  of 
Aviation  Psychology  (Rantanen,  McCarley  &  Xu,  2004): 

“To  carry  out  their  tasks,  controllers  rely  on  a  mental  picture  comprising  static 
information  of  the  airspace  layout,  the  rules  and  standard  procedures  regulating 
the  conduct  of  flight,  aircraft  flight  plan  information  and  perfonnance 
characteristics,  and  the  dynamic  traffic  situation  changing  from  moment  to 
moment  (Rantanen,  1994).  Given  the  dynamic  nature  of  ATC,  the  temporal 
aspects  of  the  controllers’  mental  picture  are  apparent,  as  are  the  temporal 
demands  of  their  tasks:  Controllers  need  to  anticipate  aircraft  trajectories  and  pilot 
intentions  well  into  the  future,  plan  their  actions,  and  then  execute  the  planned 
actions  at  a  proper  time  and  in  an  appropriate  sequence.” 

Given  this  observation,  it  is  extremely  important  that  controllers  have  all  the 

necessary  and  critical  infonnation  available  to  make  effective  decisions  in  a  timely 

manner.  This  makes  availability  of  critical  infonnation  invaluable  to  facilitate  accurate 

decision  making;  the  more  critical  information  accessed  by  controllers,  the  better  their 

ability  to  make  good  decisions. 
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The  goal  of  this  paper  is  to  study  the  impacts  of  additional  text  tool  on  controller's 
perfonnance.  In  light  of  the  aforementioned,  the  focus  question  presented  for  this  study 
is: 

Does  supplementary  computer  generated  cueing  enhances  controller  efficiency  in 
a  congested  communication  environment? 

Research  Objective 

In  the  future,  ATC  will  be  frequently  challenged  due  to  the  expected  increase  in 
the  amount  and  complexity  of  air  traffic  (Stein  and  Garland,  1993).  The  Federal  Aviation 
Administration  (FAA)  Aerospace  Forecast  for  Fiscal  Years  2012-2032  stated  that  in 
2011,  total  activity  at  FAA  en-route  centers  increased  by  1.8  percent  from  the  previous 
year  to  a  total  of  41.2  million,  the  fastest  growth  since  2005.  It  also  predicts  that  airport 
tower  operations  are  expected  to  increase  by  23  percent.  Also,  the  number  of  aircraft 
handled  at  FAA  en-route  centers,  which  separate  high  altitude  traffic,  is  expected  to 
increase  by  50  percent  (FAA,  2012). 

The  area  of  particular  interest  in  this  study  is  the  communication  between  pilots 
and  controllers,  which  is  the  foundation  of  air  traffic  control  (Hopkin,  1995).  As  the  air 
traffic  load  increases,  the  same  is  expected  to  occur  with  communication  load  in  ATC. 
Frequency  congestion  is  a  sensitive  situation  that  occurs  during  the  communication 
process  between  pilots  and  controllers  and  “has  become  a  factor  that  severely  constrains 
the  capacity  of  the  National  Airspace  System  (NAS)”  (Rantanen,  McCarley  &  Xu,  2004). 
Controllers  often  deal  with  a  congested  communication  environment  and  are  frequently 
required  to  communicate  with  several  aircraft  over  multiple  radio  frequencies.  As  a 
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result,  during  peak  loading,  it  is  common  for  the  controller  to  receive  multiple  concurrent 
communications,  each  from  a  different  aircraft,  and  it  becomes  a  challenge  to  discern 
audio  messages  received  from  multiple  pilots  simultaneously  (this  phenomenon  is  known 
as  “step-on”).  As  operators  monitor  all  frequencies  through  a  single  earpiece,  it  is 
impossible  for  the  controller  to  interpret  concurrent  communications  resulting  in 
additional  workload  as  the  controller  must  then  decipher  the  source  of  each 
communication  and  request  each  aircraft  to  repeat  the  communication  asynchronously.  In 
some  extreme  cases,  improper  or  misunderstood  communication  is  also  the  cause  of 
many  accidents  (Rantanen  et  al,  2004). 

The  proposed  method  improves  upon  the  existing  ATC  workstation  by  providing 
text  transcriptions  of  pilot's  audio  communication  to  ATCs.  The  text  transcriber 
technology  is  used  to  transcribe  all  pilots’  communication  from  different  frequencies  into 
a  single  text  box  in  the  right  side  of  the  radar  display  with  a  timestamp,  based  on  the 
actual  time  of  the  transmission  -  the  last  call  is  displayed  sequentially  as  the  last  text- 
transcription  -  as  shown  in  Figure  1,  allowing  a  controller  to  easily  read  what  is  being 
said  by  pilots  of  aircraft  under  their  control.  The  text  allows  the  controller  to  respond 
without  having  to  transmit  a  request  for  a  repeat,  wait  for  a  re-transmission  by  a  pilot, 
then  respond,  which  adds  exponentially  to  frequency  congestion.  This  design  provides 
the  controllers  with  all  the  necessary  information  in  a  usable  and  readable  format  for  the 
communication  task,  located  on  a  single  window,  displayed  in  a  sequentially  manner, 
according  to  the  sequence  of  pilots'  transmissions,  as  it  is  desired  in  the  design  of  the 
ATCS  (Cardosi  and  Murphy,  1995).  It  should  also  decrease  the  time  and  steps  needed  to 
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answer  pilots’  requests,  increasing  controller's  efficiency  and  safety  within  the  National 
Airspace  System  (NAS). 

The  objective  of  this  study  is  to  identify  possible  benefits  of  using  supplemental 
text  cueing  of  pilots’  transmissions  in  order  to  enhance  controllers’  ability  to  understand 
pilots’  requests  and  potential  improvement  on  controllers’  efficiency  and  ultimately 
guarantee  the  safety  of  the  NAS. 


Figure  1:  Radar  display  with  proposed  text  on-screen  technology 

Summary 

This  chapter  discussed  the  problems  associated  in  dealing  with  a  congested 
communication  environment.  The  following  chapter  will  review  the  literature  around 
ATC  and  cognitive  characteristics  that  may  provide  insight  into  improving  controllers’ 
performance.  Specific,  hypotheses  concerning  supplemental  text  cueing  will  be  proposed. 
Chapter  three  will  present  the  methodology  for  measuring  the  impact  of  this  technology 
on  controller  efficiency,  information  on  the  relevant  population,  and  the  experiment  data 
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collection  techniques.  Chapter  four  will  analyze  the  data  collected  from  the  participants 
during  the  experiment.  Chapter  five  will  discuss  the  results,  findings,  limitations,  and 
implications  of  this  research  effort  to  the  ATC  community  and  the  US  Air  Force,  and 
concludes  with  recommendations  for  future  research. 
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II.  Literature  Review 


Chapter  Overview 

This  chapter  contains  a  literature  review  with  detailed  information  and  research 
relevant  to  this  effort.  This  portion  of  the  writing  culminates  with  the  generation  of  five 
hypotheses  that  were  developed  to  investigate  the  research  question  listed  in  Chapter  1 . 

Description 

ATC  Environment 

The  proposed  technology  is  based  on  a  user-centered  design  approach.  This 

approach  emphasizes  that  the  system  is  expected  to  attend  to  the  users’  demands.  The 

main  idea  behind  an  user-centered  design  is  that  “if  a  product,  an  environment  or  a 

system  is  intended  for  human  use,  then  its  design  should  be  based  upon  the  characteristics 

and  needs  of  its  human  users”  (Pheasant,  1989).  According  to  the  European  Organization 

for  the  safety  of  Air  Navigation  (EUROCONTROL),  in  their  paper  regarding  Human 

Factors  (HF)  Integration  in  Future  Air  Traffic  Management  (ATM)  Systems  (2000): 

“User-centred  design  is  an  approach  to  systems  design  that  emphasizes  that  the 
purpose  of  the  system  is  to  serve  the  user.  It  is  important  to  understand  who  the 
users  are,  and  the  tasks  they  do.” 

In  order  to  study  the  impacts  of  new  technology  in  the  ATCS,  it  is  important  to 
understand  the  characteristics  of  the  environment  that  it  is  dealing  with. 

ATC  is  a  dynamic  environment  where  controllers  constantly  receive  a  large 
volume  of  infonnation  from  multiple  sources  to  monitor  changes  in  the  environment, 
make  decisions,  and  perform  effective  actions  in  a  timely  manner  (Xing  &  Manning, 
2005).  The  NAS  consists  of  a  vast  network  of  people  and  equipment  designed  to  provide 
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a  safe  and  efficient  flying  environment.  ATC  is  a  service  provided  by  ground-based 
controllers  that  work  within  the  NAS  to  direct  aircraft  on  the  ground  and  in  the  air. 
Controllers  are  responsible  for  managing  and  coordinating  the  flow  of  air  traffic  and 
ensuring  safe  flights,  arrivals,  and  departures  with  minimal  delays. 

The  role  of  controllers  is  a  critical  one.  The  ATC  system’s  intricacies  include 
thousands  of  separate  facilities  all  communicating  with  each  other  and  handling 
information  via  different  sources  (e.g.,  radar  screen  with  a  series  of  automated  visual 
cues,  paper  or  electronic  flight  progress  strips,  radio  and  interphone  communication).  Air 
traffic  controllers  have  to  deal  with  all  these  different  sources  of  information  and  control 
complex,  dynamic  and  time-constrained  traffic  situations  to  identify  and  resolve  potential 
conflict  and  risky  relationships  between  aircraft.  Many  studies  have  been  conducted  to 
assess  the  complexity  of  air  traffic  control  (Mogford,  Guttman,  Morrow,  &  Kopardekar, 
1995;  Laudeman,  Shelden,  Branstrom,  &  Brasil,  1998).  Controllers  have  to  perceive, 
analyze,  comprehend,  and  anticipate  multiple  characteristics  and  flight  paths  of  many 
aircraft  while  new  incoming  aircraft  create  new  traffic  relationships  for  evaluation.  All  of 
this  must  be  performed  in  real-time  and  for  the  purpose  of  separating  aircraft  and  issuing 
safety  alerts  (FAA,  2012). 

In  dealing  with  dynamic  and  complex  tasks  it  is  important  to  understand  how 
controllers  are  affected  by  technology  to  understand  the  impact  of  a  new  system  in  this 
environment. 
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Human  Factors  and  ATC  Communication 


Human  factors  implications  of  automation  have  been  heavily  researched  (Wiener 

and  Curry,  1980;  Wiener,  1985;  Sarter  and  Woods,  1994,  1995a,  1995b).  The 

EUROCONTROL,  in  another  paper  about  HF  case  process  (2007),  stated  that: 

“The  goal  of  HF  is  to  better  match  the  system  to  the  human,  and  the  human  to  the 
system.  Incorporating  the  wider  view  of  all  the  HF  aspects  into  the  design  and 
ongoing  operation  of  the  ATM  system  increases  efficiency,  enhances  safety,  and 
reduces  costs  in  the  long  term. 

ATM  system  design  must  be  user-centred  and  based  on  operational  requirements 
to  make  the  best  use  of  human  strengths  and  capabilities  while  compensating  to 
the  maximum  possible  extent  for  human  limitations.” 

The  use  of  new  tools  requires  that  controllers  integrate  the  interaction  demands  of 
the  new  system  into  the  management  of  their  cognitive  resources  (Bressolle,  Benhacene, 
Boudes,  &  Parise,  2000).  The  automation  should  be  carried  out  with  analysis  of  controller 
information  needs  and  an  understanding  of  the  perceptual  and  cognitive  characteristics  of 
the  controller  to  effectively  support  decision  making  (Wickens,  1998).  This  leads  to  the 
concept  of  cognitive  engineering,  which  implies  the  need  to  understand  what  really 
makes  the  task  difficult  for  the  operators  and  how  this  difficulty  impairs  human 
perfonnance  so  as  to  define  the  most  effective  aids  (Rasmussen  1986;  De  Montmollin  & 
De  Keyser,  1985;  Mancine,  Woods  &  Hollnagel,  1988). 

Controllers  deal  with  timely  decision  making  and  need  to  be  provided  with  the 
necessary  infonnation  to  make  an  effective  decision.  In  addition  to  designing  systems 
that  provide  the  operator  with  the  needed  information  and  capabilities,  it  must  be 
provided  in  a  way  that  it  is  usable  cognitively  (Endsley  &  Garland,  2000).  If  infonnation 
provided  by  the  tools  overwhelms  controllers’  cognitive  capacities,  critical  infonnation 
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could  be  either  missed  or  misinterpreted  and  put  perfonnance  at  risk  (Xing  &Manning, 
2005).  The  way  infonnation  is  presented  to  controllers  should  reduce  complexity  and 
improve  understanding  of  the  environment.  For  air  traffic  control,  complexity  reflects  the 
difficulty  to  formulate  an  accurate  mental  representation  of  the  situation,  given  many 
sources  of  infonnation  about  aircraft,  sectors,  and  flight  rules  (Xing  &  Manning,  2005). 
This  is  an  important  concept  that  needs  to  be  considered  when  designing  a  system  in  an 
ATC  environment,  and  it  is  known  as  situation  awareness  (SA). 

A  generic  model  of  the  cognitive  processes  involved  in  the  ATC’s  task,  proposed 
by  Wickens  et  al.  (1997),  illustrated  in  Figure  2,  which  states  that  SA  is  the  principal 
input  to  decision  making  and  planning.  Situation  awareness  is  a  fundamental  requirement 
for  effective  air  traffic  control  fonning  the  basis  for  controller  decision  making  and 
perfonnance  (Endsley  &  Rodgers,  1994;  Endsley  &  Smolensky,  1998).  It  can  be  said  that 
SA  is  the  operator's  internal  model  of  the  state  of  the  environment.  Based  on  that 
representation,  operators  will  make  decisions  about  the  presented  situation  (Endsley, 
1995b).  SA  is  the  result  of  the  continuous  extraction  of  environmental  information, 
integration  of  this  infonnation  with  previous  knowledge  to  form  a  coherent  mental 
picture,  and  the  use  of  that  picture  in  directing  further  perception  and  anticipating  future 
events  (Dominguez  et  al.1994).  SA  is  the  design  objective  of  systems  to  support  the 
operator’s  ability  to  obtain  the  needed  information  under  dynamic  operational  constraints 
(Endsley  &  Garland,  2000). 
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Figure  2:  Wickens  et  al.’s  (1997)  cognitive  model  of  the  controller’s  task 


The  model  in  Figure  2  shows  five  cognitive  stages  that  lies  between  events  and 
actions  involved  in  the  controller’s  task:  selective  attention,  perception,  SA,  decision 
making  and  planning,  and  action  execution.  In  the  controllers’  tasks,  actions  (on  the  right) 
are  driven  by  events  (on  the  left).  The  model  also  includes  the  influence  of  memory  in  the 
controller’s  task  where  the  immediate  memory  supports  the  SA  for  dynamic  events,  while 
the  prospective  memory  supports  actions  to  be  made  in  the  near  future.  Each  of  the  five 
processes  draws  on  knowledge  stored  in  the  structures  of  long-term  memory.  The  model 
suggests  that  controllers  selectively  perceive  events  to  develop  and  maintain  SA  of  the 
situation,  and  that  SA  is  the  principal  input  to  decision  making  and  planning,  which  may 
result  in  the  execution  of  actions  such  as  communications  and  keyboard  use  (D’ Arcy  and 
Rocco,  2001).  Controllers  respond  to  events  from  many  different  forms,  including  visual 
changes  on  the  radar  display,  such  as  the  one  proposed  by  the  additional  text  cueing.  For 
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this  reason,  based  on  this  model,  decision  aids  in  the  radar  display,  may  influence 
decision  making  and  planning  because  it  influences  the  process  of  building  and 
maintaining  SA. 

To  provide  systems  and  cues  that  better  support  the  decision  making  process  and 

enhance  situation  awareness  it  is  also  important  to  understand  how  human  processing  of 

information  occurs.  Wickens’  (2008)  multiple  resource  theory  (MRT)  4-D  model, 

illustrated  in  Figure  3,  identified  the  structural  dimensions  of  human  information 

processing  and  proposes  that  there  are  four  dimensions  (processing  stages,  perceptual 

modalities,  visual  channels,  and  processing  codes)  with  two  categorical  levels  of  each 

dimension  (i.e.  separate  resources).  According  to  the  author  (Wickens,  2002),  in  another 

paper  relating  multiple  resources  and  performance  prediction: 

“All  other  things  being  equal  (i.e.  equal  resource  demand  or  single  task 
difficulty),  two  tasks  that  both  demand  one  level  of  a  given  dimension  (e.g.  two 
tasks  demanding  visual  perception)  will  interfere  with  each  other  more  than  two 
tasks  that  demand  separate  levels  on  the  dimension  (e.g.  one  visual,  one  auditory 
task)”. 

The  theory  proposes  that  distinct  pools  of  resources  require  different  types  of 
processing.  It  is  based  on  the  assumption  that  separate  attentional  resources  (audio, 
visual,  etc.)  are  associated  with  different  processing  stages,  processing  codes,  and  sensory 
channels. 


12 


Figure  3:  Wickens’  Four-Dimensional  MRT  Model 


This  theory  suggests  that  cognitive  resources  used  to  process  information  from 
one  modality  do  not  compete  with  the  resources  used  to  process  input  from  another 
modality,  resulting  in  a  task-sharing  benefit  if  infonnation  is  distributed  across  different 
resources.  According  to  this  theory,  an  improvement  in  performance  would  be  expected 
when  different  input  modalities  are  addressed  in  a  multitask  environment  (Parasuraman 
&  Metzger,  2006). 

Multiple  Resource  Theory  has  been  heavily  explored  by  researchers  to  study  the 
effects  generated  by  the  inclusion  of  systems  or  different  cues  that  require  different 
attentional  resources  in  the  ATC  environment  (Weikert,  2002;  Hopp,  Smith,  Clegg  & 
Heggestad,  2006).  Parasuraman  &  Metzger  (2006),  in  their  study  about  effects  of 
automated  conflict  cueing  on  air  traffic  controller  perfonnance  and  visual  attention,  stated 
that  the  use  of  auditory  feedback  is  already  being  investigated  by  the  FAA  (Newman  & 
Allendoerfer,  2000).  Auditory  channels  are  not  currently  utilized  for  feedback  or  cueing 
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from  automation  tools  in  ATC  due  to  the  heavy  use  of  voice  communication  in  today’s 
ATC  operations. 

Typically  a  controller  wears  a  single-ear  headphone  in  which  they  receive  inputs 
from  many  different  audio  sources  including  but  not  limited  to;  air  to  ground  radios,  inter 
and  intra  facility  coordination  phones,  speakers  (ambient  “shout”  lines),  other  controller 
positions,  and/or  landline  telephones  including  both  direct  dial  and  party  lines.  These  are 
used  to  communicate  with  other  controllers  and  well  as  listening  to  and  transmitting  on 
(monitor)  different  frequencies.  Often  a  controller  may  monitor  as  many  as  a  dozen 
different  UHF/VHF  frequencies,  to  receive  communication  from  pilots.  Of  particular 
interest  for  this  study  is  the  situation  when  two  or  more  pilots  are  operating  on  different 
frequencies  but  being  handled  by  one  controller,  as  is  very  frequently  the  case  between 
civilians  using  VHF  and  military  aircraft  on  UHF  for  example.  The  pilots  do  not  hear 
each  other  but  the  controller  can  have  one  transmission  completely  obscured  by  the  other, 
pennitting  the  controller  to  understand  bits  and  pieces  but  not  all  of  both  communications 
leading  to  potential  misunderstandings.  "Step-ons"  as  they  are  commonly  referred  to,  are 
simply  two  or  more  users  transmitting  simultaneously  resulting  in  interference  from  the 
perspective  of  the  controller  regarding  both  transmissions. 

To  correct  this,  a  controller  follows  a  simple  but  time  consuming  and  labor 
intensive  process,  distracting  them  from  their  primary  task  usually  during  periods  of 
increased  traffic  volume  and  complexity.  Once  a  step-on  occurs  and  two  transmissions 
are  overridden,  distorted  or  obscured,  the  controller,  if  he/she  saw  which  frequencies 
transmitted,  will  isolate  at  least  one  of  them  by  de-selecting  the  transmit  function  of  the 
other  radio  channels  to  prevent  simulcasting  a  request  for  repeat.  This  involves  asking  a 
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pilot  to  repeat  and  getting  the  same  step-on  condition  again.  They  will  then  transmit  a 
"say  again"  call  to  get  one  of  the  aircraft  to  repeat  themselves.  The  pilot  will  repeat  and 
the  controller  may  not  monitor  or  "tune-out"  one  of  the  other  frequencies  to  focus  on 
what  was  previously  missed  -  a  risky  practice  during  heavy  traffic.  Once  the  controller 
gets  the  requested  infonnation,  they  will  respond  and  then  deselect  this  frequency  to  key- 
up  the  previously  de-selected  frequency  to  repeat  the  process  with  the  other  pilot.  This 
can  cost  precious  time,  and  lead  to  loss  of  situation  awareness  as  all  other  transmissions 
and  pilot  requests  go  unanswered  or  are  delayed  during  this  isolation  repeat/isolation 
repeat  sequence.  In  their  study  about  time  delays  in  the  air  traffic  communication  loop, 
Rantanen  et  al  (2004)  stated  that  “frequency  congestion  has  become  a  factor  that  severely 
constrains  the  capacity  of  the  National  Airspace  System  (NAS)”. 

Additionally,  Drullman  and  Bronkhorst  (2000),  in  their  study  about  multichannel 
speech  intelligibility  and  talker  recognition  using  monaural,  binaural,  and  three- 
dimensional  speech  stated  that  speech  communication  is  degraded  when  speakers  talk 
over  one  another.  While  ATC  does  not  use  ambient  open-air  speakers,  but  rather  in-ear  or 
over  the  ear  mono-speaker  headphones,  Drullman  and  Bronkhort's  results  are  still 
pertinent  for  detennining  the  effectiveness  of  the  communication  while  obscured  by 
multiple  transmissions.  The  percentage  of  words  and  sentences  understood  correctly  in 
the  study  decrease  as  the  number  of  competing  talkers  increases.  Figure  4  shows  the 
mean  percentage  of  words  (panel  A)  and  sentences  (panel  B)  correct  for  the  different 
presentation  modes  (e.g.  monaural,  binaural,  and  3D  with  individual  or  general  head- 
related  transfer  functions)  as  a  function  of  the  number  of  competing  talkers.  As  expected 
by  the  authors,  the  scores  decrease  as  the  number  of  competing  talkers  increases. 
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Figure  4:  DruIIman  and  Bronkhorst  speech  intelligibility  results 


Given  the  potential  for  decreased  understanding  arising  from  simultaneous 
speakers  and  the  multitask  environment  of  controllers,  another  source  of  attention, 
according  to  the  MRT  Model  in  Figure  3,  would  be  useful  to  explore.  Therefore,  text 
cueing  may  be  particularly  useful,  in  the  sense  that  it  adds  other  source  of  attention  to 
decrease  audio  congestion  and  potentially  improve  controllers’  efficiency. 

The  role  of  computer  generated  text  cueing 

A  European  study  (Nijhuis  and  North,  1998)  called  the  “Role  of  the  Human  in  the 
Evolution  of  ATM”  provided  a  framework  within  which  each  automation  concept  may, 
in  principle,  be  categorized.  The  report  stated  that  “full  automation”  and  “the  controller 
as  a  supervisor  of  the  system”  are  not  suitable  because  of  the  negative  impact  on  safety. 
The  report  also  stated  that  “cognitive  tools”  seem  to  be  the  most  promising  method  for 
improving  ATC  effectiveness.  The  report  defined  cognitive  tools  as  “machine  assistance 
for  organizing  controller  tasks  based  on  typical  controller's  heuristics”.  To  improve 
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effectiveness,  automation  should  be  driven  by  a  human-centered  approach.  Automation 
should  assist  and  augment  the  capabilities  of  the  controller. 

In  a  critical  environment,  where  information  must  be  captured,  processed  and 
decisions  must  be  made  in  a  timely  manner,  and  where  errors  are  critical  factors,  the  level 
of  automation  must  be  carefully  limited.  Technology  must  be  used  to  support,  not 
replace,  the  human  in  the  decision  making  process  in  ATC.  Computer  systems  are  limited 
in  technical  knowledge  and  are  not  as  versatile  as  the  human  expert  (Will,  1991). 
Computers  should  not  make  decisions  for  the  controller.  Instead,  to  maximize  its  impacts, 
computers  should  provide  assistance  in  making  those  decisions. 

Willems  and  Truitt  (1999)  studied  the  effect  of  moving  a  controller  from  the 
current  active  control  to  a  monitoring  position.  The  author  concluded  that  controller 
situation  awareness  was  lower  under  monitoring  conditions;  however  controllers 
perceived  that  their  situation  awareness  did  not  change  between  active  control  and 
passive  monitoring,  which  warrants  even  more  careful  examination. 

Studies  have  shown  that  operators  have  a  better  mental  model  or  awareness  of  the 
system  state  when  they  are  actively  involved  in  creating  the  state  of  the  system  than  when 
they  are  passively  monitoring  the  actions  of  another  agent  or  automation  (Endsley,  1996; 
Endsley  &  Kiris,  1995).  Several  studies  have  shown  that  conflict  detection  perfonnance 
and  situation  awareness  were  reduced  and  mental  workload  increased  under  simulated 
free  flight  (FF)  conditions,  where  there  is  a  transfer  of  responsibility  for  separation 
between  aircraft  from  controllers  to  pilots.  (Castano  &  Parasuraman,  1999;  Corker, 
Fleming,  &  Lane,  1999;  Endsley,  Mogford,  Allendoerfer,  Snyder, &Stein,  1997;  Endsley 
&  Rogers,  1998;  Galster,  Duley,  Masalonis,  &  Parasuraman,  2001;  Metzger  & 
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Parasuraman,  2001).  These  investigations  provided  the  first  empirical  evidence  of  the 
effects  of  future  ATM  concepts  on  controller  perfonnance  and  pointed  to  the  passive 
monitoring  role  (Metzger  &  Parasuraman,  2001)  as  factors  contributing  to  reduced 
controller  perfonnance  (Metzger  &  Parasuraman,  2005). 

The  addition  of  text  cueing  is  proposed  to  be  an  optional  tool  for  air  traffic 
controllers.  They  should  be  able  to  choose  to  display  it  or  minimize  it  on  screen  to 
increase  their  own  efficiency.  The  role  of  the  text  is  to  supplement  the  audio 
communication  from  pilots  and  not  substitute  it,  provide  redundancy  of  audio 
communications  received  from  pilots  to  provide  better  frequency  management,  and 
reduce  superfluous  transmissions  to  reduce  frequency  congestion,  which  will  also  reduce 
response  time. 

The  text  as  visual  infonnation  will  co-occur  with  the  audio  transmission,  which 
shall  increase  the  probability  that  at  least  one  signal  will  be  detected.  The  advantage  of 
adding  a  visual  component  to  audio  is  by  providing  redundancy  and  to  facilitate  the 
understanding  the  audio  communication.  Redundancy  is  a  useful  technique  that  can  be 
provided  to  decrease  or  even  prevent  information  loss.  Studies  have  shown  that  speech 
intelligibility  performance  generally  improves  as  complexity  increases  and  contains 
higher  degree  of  redundancy  and  more  contextual  information  (Rash,  Russo,  Letowski, 
and  Schmeisser,  2009).  Further  voice  communication  is  transient  while  text  is  persistent, 
pennitting  ATC  personnel  to  choose  to  access  the  infonnation  at  the  earliest  convenient 
time,  as  opposed  to  disrupting  every  other  task  to  insure  receipt  of  auditory  information. 

Therefore,  in  the  light  of  the  aforementioned,  the  text  cueing  is  hypothesized  to 
improve  frequency  congestion  and  ultimately  controllers’  efficiency  in  the  sense  that  it 
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will  reduce  the  need  for  say-again  transmissions,  because  controllers  will  be  able  to  better 
comprehend  pilot’s  transmission  with  the  additional  text  on-screen. 

HI:  Supplementary  text  transcription  of  pilots  ’  communication  will  increase 
controllers  ’  comprehension  of  pilots  ’  transmissions. 

In  developing  a  system  that  enhances  controllers’  capabilities  on  the 
communication  task,  characteristics  of  the  technology  shall  be  considered  in  order  to 
analyze  the  impact  of  the  media  on  controllers’  perfonnance. 

Media  Richness  Theory  proposes  that  each  channel  has  its  own  characteristics 
that  determine  its  capacity  of  reproducing  rich  infonnation  (Daft  and  Lengel,  1984). 
According  to  this  theory,  specific  messages  should  be  communicated  on  the  appropriate 
channel  that  provides  the  sufficient  media  richness  capacity.  Carlson  and  Zmud  (1994) 
reformulated  the  Media  Richness  Theory,  and  proposed  the  Channel  Expansion  Theory 
(CET).  The  authors  stated  that  media  richness  is  seen  not  only  based  on  the  channel 
characteristics  for  itself,  but  to  be  more  a  perception  of  the  user  based  on  familiarity  and 
experience  with  the  medium,  and  less  a  characteristic  intrinsic  to  the  channel. 

Based  on  this  theory,  an  individual’s  perception  about  the  richness  of  the  channel 
may  affect  how  the  individuals  improve  their  performance  during  the  communication 
task  using  the  channel.  Additional  text  on-screen  is  the  media  presented  to  controllers  in 
order  to  enrich  their  perfonnance  on  the  communication  task.  Familiarity  and  experience 
with  the  proposed  technology  may  influence  controllers’  ultimate  perfonnance  with  the 
text  cueing. 

Therefore,  it  is  expected  that  text  cueing  shall  be  perceived  as  a  rich  media  by  the 
participants  that  are  already  familiar  with  the  technology  more  than  the  controllers  that 
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are  not  so  experienced  with  the  on-screen  text.  Familiarity  is  expected  to  influence 
controllers’  perfonnance  with  the  technology. 

Memory  recall 

Short-term  memory  accuracy  has  been  a  fundamental  talent  required  by 
controllers  to  base  decisions  and  actions  to  ensure  safety.  It  is  an  important  characteristic 
in  the  decision  making  process  in  ATC  because  the  scene  is  very  dynamic,  with  rapid 
changes. 

Decision  making,  in  the  context  of  air  traffic  control  systems,  is  a  continuing 
activity  that  utilizes  the  controller's  comprehensive  understanding  of  information  at 
several  levels  of  detail,  together  with  the  controller's  knowledge  of  rules,  procedures  and 
instructions  and  their  permissible  flexibility  (Hopkin,  1995). 

Controllers  are  decision  makers  in  a  dynamic  environment  involving  many 
factors;  constant  updating  of  relevant  information,  and,  most  of  the  time,  conflict 
detection  and  avoidance.  Controllers  must  address  the  sometimes  conflicting  goals  of 
safety  and  efficiency,  “through  an  intricate  series  of  procedures,  judgments,  plans, 
decisions,  communications,  and  coordinated  activities”  (Wickens,  Mavor,  &  McGee, 
1997). 

Numerous  studies  have  been  done  and  different  systems  have  been  researched  to 
enhance  the  efficiency  of  the  ATCS  (Weber  et  al,  2007;  Couluris,  Schleicher,  and  T. 
Weidner,  1998).  Some  of  the  new  proposals  and  concepts,  such  as  free  flight  (RTCA, 
1995),  distributed  air  ground  traffic  management  (NASA,  1999),  Automatic  Dependent 
Surveillance-Broadcast  (McCallie,  2011),  and  next  generation  air  transportation  system 
(Joint  Program  Development  Office,  2004)  will  change  the  role  of  air  traffic  controllers 
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altogether.  Pilots  and  airline  dispatchers  will  receive  greater  freedom  to  choose  speed, 
heading  and  altitude  (Parasuraman  and  Metzger,  2006).  Controllers  must  be  able  to 
remain  in  control  of  such  complex  and  dynamic  systems,  which  requires  that  the 
controller  maintain  a  high  level  of  situation  awareness. 

According  to  Endsley  (1995b)  there  is  a  strong  relationship  between  SA  and 
memory.  In  her  work,  the  author  reviewed  various  methods  for  measuring  SA  and  stated 
that  “SA,  composed  of  highly  relevant,  attended  to,  and  processed  information,  should  be 
most  receptive  to  recall”.  According  to  Endsley  a  participant’s  SA  can  be  accessed  by 
measuring  recall  (Gronlund  et  al,  1997).  In  this  sense,  the  importance  of  improving  short¬ 
term  memory  recall  is  directly  related  to  the  improvement  of  situation  awareness. 

According  to  Tindall-Ford,  Chandler,  and  Sweller  (1997),  the  use  of  dual-mode 
presentation  techniques  (e.g.,  auditory  text  and  visual  diagrams)  with  students  can  result 
in  superior  learning  to  equivalent,  single-modality  formats  (e.g.,  visual  text  and  visual 
diagrams).  According  to  the  authors,  this  modality  effect  may  be  ahributed  to  an  effective 
expansion  of  working  memory. 

With  the  use  of  text  transcription  on  screen,  controllers  should  be  able  not  only  to 
listen,  but  also  read  what  was  said,  allowing  the  participants  to  use  two  sensory 
modalities  (e.g.,  auditory  and  visual)  instead  of  just  one  (e.g.,  auditory),  according  to  the 
MRT  model.  The  information  presentation  will  be  split  between  auditory  and  visual 
display,  in  an  attempt  that  neither  mode  will  be  overused,  as  desired  for  the  design  of 
ATCS  (Cardosi  and  Murphy,  1995).  As  it  occurs  in  the  learning  processing  with 
students,  when  controllers  listen  and  read  pilots’  transmission  it  is  hypothesized  to  also 
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improve  memory  recall.  In  this  sense,  text  cueing  is  hypothesized  to  improve  controller’s 
recall  of  what  was  seen  and  heard  within  the  air  traffic  scene. 

H2:  Supplementary  text-cueing  will  improve  memory  recall  on  pilots  ’ 
transmissions 

Design  Considerations 

Design  considerations  were  crucial  to  include  the  text  on  the  radar  screen  to 
increase  controller’s  performance  and  reduce  potential  distraction  caused  by  the  new  text 
on  screen,  since  a  user-centered  design  approach  was  adopted  for  the  proposed 
technology. 

Distraction  is  a  psychological  error  mechanism  (PEM)  that  is  known  to  affect 
perfonnance  (Shorrock  and  Kirwan,  2002).  In  their  paper  describing  a  human  error 
identification  (HEI)  technique  called  TRACEr,  a  technique  for  the  retrospective  and 
predictive  analysis  of  cognitive  errors  in  ATC,  the  authors  stated  that  distraction  can 
affect  different  cognitive  domains,  such  as:  perception  and  vigilance  (i.e.  errors  in  visual 
detection  and  visual  search,  and  errors  in  listening);  memory  (i.e.  forgetting  or 
misrecalling  temporary  or  longer-term  information,  forgetting  previous  actions,  and 
forgetting  planned  actions);  and  action  execution  (i.e.  actions  or  speech  performed  not  as- 
planned).  For  this  reason,  potential  causes  of  distraction  in  ATC  should  be  carefully 
considered  in  the  design  of  new  aids  to  ensure  that  controllers’  perfonnance  will  not  be 
impaired. 

Today  controllers  are  not  experienced  with  supplementary  text  displayed  on  the 
side  of  the  radar  screen.  Ensuring  safety  is  a  priority  for  the  design  of  ATCS.  Although 
the  text  box  was  carefully  designed  to  reduce  distraction  such  that  safety  would  not  be 
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compromised,  this  factor  was  measured  as  a  self-reported  question  to  check  the  potential 
distraction  caused  by  the  text  on  radar  screen. 

H3:  Supplemental  text  box  will  increase  controller's  distraction 

Some  design  choices  were  also  considered  in  order  to  decrease  potential 
distraction  and  increase  controller  performance  through  the  addition  of  text.  Parasuraman 
and  Metzger  (2006)  suggested  that  the  controllers’  primary  display  is  the  radar  display 
and  this  is  also  where  critical  events  would  most  likely  unfold  and  have  to  be  detected. 
Visual  attention  might  be  directed  there  by  default.  If  the  visual  alert  indicating  a  conflict 
is  integrated  on  the  display,  chances  are  low  that  it  will  be  missed  (Parasuraman  & 
Metzger,  2006). 

Considering  some  changes  caused  by  all  innovations  proposed  for  ATCS  and  the 
increasingly  congested  environment,  controllers  will  need  tools  to  enhance  perfonnance 
and  SA  while  monitoring  the  air  traffic.  Text  cueing  could  improve  performance  and  save 
controller’s  time,  making  more  time  available  to  monitor  the  radar  display. 

Also,  Murray  &  Caldwell,  1996  found  that  the  number  of  displays,  in  which 
participants  should  detect  and  identify  objects,  had  a  great  impact  on  participant 
perfonnance.  When  increasing  the  number  of  displays  that  need  to  be  monitored  the 
response  time  of  the  operators  was  reduced.  Therefore,  integrating  the  text  in  the  radar 
display  will  reduce  time  spent  to  monitor  different  displays,  which  will  better  fit  with 
controllers’  visual  attention  when  integrated  on  the  radar  display. 

The  text  box  that  transcribes  the  audio  communication  was  included  on  the  radar 
display  to  be  more  effective  and  reduce  time  spent  moving  from  one  display  to  another 
and  reduce  time  where  controllers  are  not  looking  to  the  main  screen  (radar  display). 
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Regarding  positioning,  it  was  also  considered  that  the  right  side  of  the  main  display  may 
be  more  effective.  Cultural  aspects  should  be  considered  while  developing  an  interface 
that  may  be  more  useful.  People  tend  to  scan  the  elements  in  an  interface  similar  to 
general  reading  (left  to  right;  top  to  bottom  -  Western  cultures).  Western  culture’s 
interfaces  should  have  a  left  to  right  flow  of  information  (Russo  and  Boor,  1993).  The 
most  important  infonnation  (radar  screen)  should  be  displayed  in  the  left  and  the  decision 
aid  was  included  in  the  right. 

The  text  box  was  located  on  the  right  side  of  the  main  display  since  the  additional 
text  must  not  block  or  obstruct  the  radar  display  which  provides  essential  ATC 
information  (Cardosi  and  Murphy,  1995). 

Additionally,  the  text  box  on  the  main  display  is  hypothesized  to  facilitate  the 
communication  task.  The  text  should  reduce  complexity  because  controllers  will  not  need 
to  follow  too  many  steps  to  understand  and  answer  pilots’  transmissions.  The  new 
technology  should  allow  them  to  easily  identify  what  was  said  and  directly  answer  the 
requests. 

H4:  Additional  text-cueing  will  reduce  perceived  difficulty  of  the  communication 
task  between  controllers  and  pilots 

Summary 

This  research  effort  proposes  that  the  text  cueing  shall  enhance  controller’s 
efficiency  in  a  congested  communication  environment.  Controllers  that  are  more 
experienced  and  familiarized  with  text  on  screen  will  perfonn  better  than  those  that  are 
not  used  to  the  proposed  technology.  Further  it  is  hypothesized  that  the  proposed 
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technology  will  reduce  the  need  of  say  again  transmissions  for  pilot’s  and  improve 
controllers’  short-term  memory  recall,  which  will  also  reduce  the  perceived  difficulty  of 
the  communication  task.  A  summary  of  the  research  hypotheses  is  provided  in  Table  1. 

Table  1:  Research  Hypotheses 
Hypothesis 


HI 

Supplementary  text  transcription  of  pilots’  communication  will  increase  controllers’  comprehension  of  pilots’ 
transmissions 

H2 

Supplementary  text-cueing  will  improve  memory  recall  on  pilots’  transmissions 

H3 

Supplemental  text  box  will  increase  controller's  distraction 

H4: 

Additional  text-cueing  will  reduce  perceived  difficulty  of  the  communication  task  between  controllers  and 
pilots 

These  hypotheses  were  developed  to  verify  the  potential  of  the  proposed 
technology.  It  is  upon  the  hypotheses  proposed  in  this  chapter  that  the  experiment 
methodology  in  Chapter  3  is  based  to  measure  the  impact  of  additional  text  cueing  on 
controllers’  communication  task 
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III.  Methodology 


Chapter  Overview 

The  preceding  chapters  identified  the  need  for  and  the  potential  of  supplementary 
text  cueing  technology  in  the  ATC  environment.  The  hypotheses  presented  suggest  that 
voice-to-text  transcriptions  will  enhance  controllers’  perfonnance  in  their  communication 
task  with  pilots  by  increasing  comprehension  of  pilots’  transmissions. 

This  chapter  will  present  the  methodology  used  to  investigate  the  research 
hypotheses  proposed  in  Chapter  2.  Included  are  outline  descriptions  of  the  research 
design,  population  under  study,  survey  instrument,  data  collection,  pilot  study,  and  the 
actual  experiment. 

Relevant  Population 

This  research  focuses  on  the  whole  air  traffic  control  population.  For  the  purpose 
of  this  study,  the  sample  population  was  concentrated  to  a  military  training  installation  at 
Kessler  Air  Force  Base,  MS,  where  the  experiment  was  conducted  using  thirty-five 
personnel,  including  instructors  and  students.  Participation  was  voluntary  and  data  was 
collected  through  a  one  week  period.  All  the  participants  were  familiar  with  the  air  traffic 
route  and  with  the  software  used  in  the  experiment.  To  avoid  compromising  the  study, 
participants  were  asked  not  to  share  the  details  of  the  study  or  talk  about  the 
questionnaires  with  others  until  the  end  of  the  data  collection  period. 
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Research  Design 

To  determine  if  support  for  the  hypotheses  identified  in  the  previous  chapter 
exists,  an  experiment  was  conducted  which  could  both  measure  perceptions  and  test  for 
efficiency  improvement  as  an  interactive  measure  during  the  experiment.  The  experiment 
was  designed  as  a  within  subjects  study  to  account  for  leaming/sequence  bias.  According 
to  Erlebacher  (1997),  a  within  subject  approach  results  in  an  increased  statistical  power. 
Keren  stated  that  a  within-subjects  study  allows,  "...the  exclusion  of  individual 
differences  resulting  in  a  higher  degree  of  sensitivity  to  treatment  effects"  (1992). 
Furthennore,  the  exposure  sequence  was  assigned  randomly,  mitigating  potential  for 
learning  bias  and  order  bias  and  further  strengthening  the  design  for  optimum 
management  of  participant  pool  and  highest  statistical  yield. 

All  the  participants  were  given  a  participant  number,  by  which  they  were 
identified  throughout  the  experiment.  Participants  were  provided  a  brief  training  period, 
and  then  provided  control  and  experimental  conditions.  Every  participant  was  their  own 
control  baseline.  The  exposure  sequence  was  assigned  randomly,  mitigating  potential  for 
learning  bias. 

A  timeline  that  details  the  experimental  sequence  and  standard  steps  was 
developed  and  exactly  followed  by  the  researchers  to  insure  the  same  conditions  and 
procedures  were  followed  with  all  the  participants.  See  APPENDIX  D  for  the  complete 
timeline  procedure  used  for  this  experiment. 

The  three  simulated  scenarios  were  presented  for  the  participants:  training,  control 
and  experimental  scenarios.  The  training  was  the  first  scenario,  followed  by  a  training 
questionnaire.  To  ensure  the  validity  of  the  measurements,  a  treatment  set  provided  to  the 
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subjects  consisted  of  a  training  session  to  familiarize  them  with  the  experiment  flow  and 
their  role  during  the  experiment.  It  offered  no  challenge  to  the  participants  and  the  data 
obtained  from  the  training  questionnaire  were  not  analyzed.  Participants  were  not 
presented  to  the  new  technology  until  the  experimental  scenario.  After  the  training,  the 
control  and  experimental  scenarios  were  assigned  to  the  participants  in  random  order. 
Participants  with  odd  numbers  had  the  control  condition  first  and  participants  with  even 
numbers  had  the  experiment  condition  first. 

Data  was  collected  during  a  one-week  period,  during  the  morning  and  afternoon, 
in  order  to  access  as  many  participants  as  possible  in  a  limited  time.  On  all  questionnaires 
handed  to  the  participants,  they  were  asked  to  provide  their  participant  number  to  track 
individual  performance  characteristics  throughout  the  experiment  tasks  without  linking 
any  other  personal  identification  other  than  the  infonnation  provided  in  the 
questionnaires.  The  participant  number  was  provided  to  the  individuals  as  they  arrived  at 
the  experiment  location. 

Scenario  Description 

For  the  experiment,  3  scenarios  were  recorded:  training,  control  and  experimental. 
The  training  scenario  was  developed  to  allow  the  controllers  to  become  familiar  with  the 
experimental  process  and  their  role  in  the  experiment.  However,  participants  were  not 
presented  with  the  new  technology  until  the  experimental  scenario.  The  control  scenario 
was  used  as  a  baseline  to  compare  controllers’  performance  and  did  not  included  any  text 
boxes,  only  a  standard  radar  screen  that  controllers  would  already  be  familiar  with.  The 
experimental  scenario  had  the  text  box  with  the  transcription  of  audio  communication, 
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and  a  screen  capture  was  shown  to  each  participant  immediately  before  the  scenario  to 
explain  how  the  text  would  appear  on  the  screen. 

All  the  scenarios  used  for  this  study  were  previously  recorded  as  movies  that  did 
not  allow  controllers’  interaction.  This  attribute  reduced  changes  on  the  scope  between 
participants  allowing  comparison  of  subjects’  perfonnance.  The  scenarios  were  designed 
to  be  as  realistic  as  possible  in  order  to  test  the  hypotheses  of  interest.  Also,  the  control 
and  experimental  scenarios  were  developed  to  be  equally  complex  in  order  to  provide 
reliable  comparison  between  controllers’  perfonnance  regarding  just  the  addition  of  the 
technology.  The  control  and  experimental  scenarios  included  8  aircraft  each,  1  of  which 
was  a  departure  and  1  an  emergency,  see  Figures  5  and  6.  A  total  of  13  pilot 
transmissions  were  recorded  for  each  scenario. 

For  the  purpose  of  this  study,  a  100  %  accurate  transcription  of  the  audio 
communication  was  assumed.  The  objective  of  the  study  is  to  collect  initial  data  of 
controllers’  perfonnance  and  perceptions  of  the  advantages  of  the  additional  text,  and  not 
to  test  the  technology  itself. 
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Figure  5:  Control  Scenario  -  Radar  Scope 


Figure  6:  Experimental  Scenario  -  Radar  Scope 
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Permission  to  conduct  research 


This  experiment  involved  the  performance  of  Air  Force  personnel  and  conducted 
experimentation  in  accordance  with  human  experimentation  requirements  (AFI  40-402). 
An  exemption  was  sought  to  use  Department  of  Defense  personnel  as  volunteers  for 
additional  text  cueing  research.  This  exemption  was  granted  by  the  Air  Force  Institute  of 
Technology  Institutional  Review  Board  (AFIT  IRB).  Additional  authorization  to  access 
active  duty  US  Air  Force  and  DoD  Civilian  controllers  granted  by  USAF  ATC  Career 
Field  Manager  and  Technical  School  Commander.  See  Appendixes  A  and  B  for 
authorization  letters. 


Equipment 

To  develop  the  ATC  simulated  scenarios  and  proceed  with  the  experiment  several 
equipment  items  including  software  and  hardware  were  used  and  are  described  below: 

1.  Lenovo  Thinkpad  T420i  ,  64  bit,  intel  core  i3-2310M  processor,  Windows  7 
Enterprise  OS,  4.00GB  memory,  300  GB  HD.  It  had  Signal  e  MMC  softwares 
installed  that  were  used  to  develop  and  record  the  ATC  scenes. 

2.  SIGNAL  Air  Traffic  Control  Simulation  Program  software.  Authorized  use  by 
FAA  to  develop  and  record  air  traffic  scenarios 

3.  Multi-Modal  Communication  (MMC)  System.  Authorized  use  granted  by  the 
inventor.  Software  used  to  transcribe  the  text 

4.  HP  computer  with  MMC  software  used  to  record  audio  and  it  also  had  the 
clickcounter  software  installed 

5.  Clickcounter  Software  freeware  mouse  click  tracking  program  used  during  the 
experiment  to  track  the  number  of  clicks  for  each  participant. 

6.  Microsoft  wireless  mobile  mouse  3500  (2).  Used  by  the  participants  for  subjects’ 
real  time  inputs  and  used  by  the  researcher  to  load  the  scenes  on  the  monitor. 

7.  Epiphan  DVI2USB  Duo  External  DVI  Capture  box  device.  It  was  used  to  record 
the  simulated  ATC  scenarios  from  the  originating  machine  (Lenovo  laptop)  to  the 
Receiver  (Dell  laptop) 
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8.  Dell  precision  M4600  workstation,  Windows  Professional  7,  64  bit,  250  GiG  HD, 
16  GB  DDR3  SDRAM,  NVIDIA  3D  suite,  NVIDIA  quadro  1000M  discrete 
graphics  card,  intel  core  i7  extreme  processor.  It  was  used  to  capture  the  ATC 
scenes  and  it  was  connected  to  the  main  display  that  was  viewed  by  the 
participants.  It  had  NVIDIA  3D  software  installed. 

9.  StarTech  display  port  to  DVI  Dual  link  Active  Converter  cable-  USB  powered.  It 
was  used  to  connect  the  Dell  computer  to  the  ASUS  monitor  to  facilitate 
operating  NVIDIA  3D  active  shutter  display. 

10.  ASUS  VG236  23.5"  3D  120Hz  LCD  display  monitor  (commercial). 

1 1 .  NVIDIA  3D  software  used  to  load  the  ATC  scenes  that  were  showed  for  the 
participants 

12.  3D  vision  USB  Controller/IR  Emitter.  It  is  a  needed  component  to  NVIDIA 
software 

13.  Headphone:  Beyerdynamic  USB  connected  MMX1  over  the  ear  headphones 


All  the  scenarios  were  developed  using  the  Signal  software,  also  the  MMC 
software  for  the  text  transcription.  They  were  recorded  from  the  Lenovo  computer  using 
the  capture  box  to  the  Dell  computer,  which  was  connected  to  the  ASUS  Monitor.  The 
scenarios  were  showed  for  the  participants  on  the  Monitor  using  the  NVIDIA  software 
installed  on  the  Dell  computer. 


Experiment 

The  experiment  was  designed  to  follow  the  research  design  and  experiment 
timeline,  see  Appendix  D.  The  workstation  that  participants  interacted  with,  see  Figure  7, 
consisted  of  a  display,  which  showed  the  radar  scenarios,  a  mouse  and  a  headphone.  The 
researchers’  workstation,  see  Figure  8,  consisted  of  two  computers:  researcher  1 
interacted  with  the  computer  that  had  the  recorded  scenarios  that  were  loaded  to  the 
participants’  display;  and  researcher  2  interacted  with  the  computer  that  had  the  mouse 
click  software  installed  to  track  the  number  of  mouse  clicks. 
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Figure  7:  Participants'  workstation 


Figure  8:  Experimental  workstation 
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The  SIGNAL  software,  see  Figure  9,  provided  familiar  training  simulations  using 
existing  training  scenarios  developed  at  the  technical  school  and  incorporating  their 
imaginary  airfield,  "Canyon  Approach".  By  taking  existing  scenarios  and  modifying  them 
to  exacerbate  the  potential  for  "step-on"  radio  transmissions  by  injecting  realistic  aspects 
of  aircraft  traffic  flow  and  event  sequencing,  we  were  able  to  develop  scenarios  that  were 
realistic  and  familiar  to  the  participants  by  using  aircraft  call  signs,  types,  airspace 
parameters  and  runways  familiar  to  them  in  their  current  training  environment.  This 
further  eliminated  the  impact  of  minimal  pre-exposure  training,  allowing  the  participants 
to  focus  on  the  conflicts  and  dynamic  aspects  of  the  scenes  without  unneeded  distraction 
having  to  overcome  any  unfamiliarity  with  unique  types,  call  signs  and  airspace/airfield 
characteristics.  Communications  between  pilots  and  controllers  were  simulated  using  pre¬ 
recorded  pseudo-pilots;  the  controllers'  responses  were  not  recorded  allowing  the 
observer's  the  opportunity  to  develop  their  own  mental  responses  and  plans-of-action 
eliminating  any  bias  a  pre-recorded  "pseudo-controller"  input  may  have  caused. 
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Figure  9:  SIGNAL  Software 

MMC  software  is  a  Multi  Modal  Communication  System  that  was  developed  by 
the  71 1th  Air  Force  Research  Laboratory’s  Battlespace  Acoustic  Branch,  see  Figure  10. 
MMC  suite  is  an  Integrated  Network-  Centric  Communication  Management  Suite  that 
provides  operators  with  net-centric,  collaborative  tools  to  accurately  monitor  multiple 
communication  channels  and  make  quick  and  effective  decisions  (Finomore  et  al,  2010). 
It  is  a  complex  communication  system  that  captures,  records,  transcribes  and  displays 
radio  and  chat  communications  to  the  operator  with  the  goal  of  alleviating  the  workload 
and  errors  associated  with  intensive  communication  environments  (Finomore  et  al., 
2011),  providing  intuitive  and  immediate  accessibility  to  communication  data. 
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Figure  10:  MMC  System  Interface 


A  limited  and  modified  version  of  the  MMC  was  used  in  this  study  to  provide  the 
on-screen  speech- to-text  transcription  of  all  pilots’  communication  from  different 
frequencies  in  a  single  text  box  in  the  right  side  of  the  radar  display,  allowing  a  controller 
to  easily  read  what  is  being  said  by  pilots  of  aircraft  under  their  control. 

Participants  were  briefed  before  starting  the  experiment  about  each  section  of  the 
simulation.  All  participants  received  a  consent  form,  see  Appendix  C,  which  they  signed 
and  stated  that  they  voluntarily  agreed  to  participate  in  this  study.  All  the  scenarios 
occurred  in  a  dark  room,  consistent  with  an  actual  air  traffic  control  facility.  A  pre¬ 
questionnaire  was  used  to  collect  some  demographics,  see  APPENDIX  E.  Participants 
were  told  that  the  scenarios  would  be  stopped  after  a  certain  time  and  that  they  would  be 
asked  detailed  questions  about  what  they  saw  and  heard.  Controllers  were  asked  to 
actively  monitor  the  scene  and  to  be  able  to  respond  to  pilots’  transmissions  as  they 
normally  would  because  that  would  be  most  beneficial  for  the  study. 
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The  first  exposure  was  always  the  training  scenario,  see  APPENDIX  F.  After  the 
training  scenario,  a  training  questionnaire,  see  APPENDIX  G,  was  handed  to  the 
participants  with  questions  about  what  they  just  saw  and  heard. 

After  the  training,  the  control  and  experimental  scenarios,  see  Figures  5  and  6, 
were  assigned  to  the  participants  in  different  orders.  After  each  3  minute  scene,  the 
display  was  blanked  and  a  questionnaire  was  handed  to  the  participants.  A  paper  copy  of 
the  last  minute  of  the  respective  scene  was  provided  to  the  participants  with  a  limited 
radar  scope,  including  just  the  position  and  the  call  signs  of  the  aircraft.  This  was 
provided  to  yield  a  memory  aid  to  help  controllers  recall  the  scene. 

Measurements  Development 

All  measurements  were  developed  to  accurately  evaluate  the  research  hypotheses. 
A  total  of  3  air  traffic  scenarios  were  presented  to  the  participants.  After  each  scenario,  a 
questionnaire  was  handed  to  the  participants.  Again,  participant  numbers  were  used  to 
mark  and  compare  individual  perfonnance  and  perceptions  about  the  technology. 

The  measure  utilized  for  the  first  hypothesis  was  the  number  of  mouse  clicks.  A 
free  open  software  application  named  “Click  Counter”  was  installed  on  a  computer  with  a 
mouse  connected  that  was  used  by  the  participants.  One  mouse  click  by  the  participants 
meant  the  need  for  a  “SAY  AGAIN”  transmission.  Participants  were  briefed  to  carefully 
monitor  and  listen  for  the  pilots’  transmissions  during  the  3 -minute  air  traffic  scene.  They 
were  asked  to  monitor  the  scene  as  if  they  needed  to  answer  all  requests  from  pilots  as 
occurs  in  a  regular  air  traffic  control  task,  but  they  should  interact  by  just  clicking  the 
mouse  one  time  if  they  feel  that  a  “SAY  AGAIN”  transmission  was  necessary  to 
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completely  understand  each  pilot  transmission  and  be  prepared  to  accurately  answer. 
During  the  experimental  scenario,  where  controllers  were  presented  with  the  text 
transcription  of  pilots’  transmissions,  participants  were  asked  to  utilize  the  new 
technology  to  the  maximum  extent  possible,  but  if  they  felt  that  a  “SAY  AGAIN” 
transmission  was  still  necessary  to  be  able  to  fully  understand  and  answer  pilots’ 
transmissions,  they  should  click  the  mouse. 

As  subjects  were  submitted  to  the  control  and  experimental  treatments,  the 
comparative  aspect  of  this  study  between  both  groups  was  expected  to  identify  that  the 
need  for  “SAY  AGAIN”  transmissions  with  the  new  technology  decreased,  therefore, 
potentially  decreasing  frequency  congestion  and  decrease  the  time  to  understand  and 
answer  pilots’  transmissions. 

For  the  second  hypothesis,  a  query  technique  was  used  to  measure  recall. 
Currently,  there  is  no  general  agreement  upon  a  correct  methodology  to  measure  SA 
(Gronlund  et  al,  1997),  but  the  query  technique  is  the  most  commonly  used  method 
(Adams,  Tenney,  and  Pew,  1995).  This  method  measures  what  the  participants  recall 
from  memory.  Using  this  method,  study  participants  engaged  in  a  task  simulation,  which 
was  temporarily  suspended  after  several  minutes.  When  the  display  blanked,  the 
participant  was  required  to  answer  a  series  of  questions  about  what  they  just  saw. 

In  this  study,  query  technique  was  used  to  measure  short-tenn  memory  recall  after 
the  controllers  were  exposed  to  a  3  minute  air  traffic  scene.  In  his  study  on  measuring 
controllers’  memory,  Gronlund  et  al  (1997)  stated  that  controllers  relied  on  spatial 
information  to  remember  large  quantities  of  information.  That  is  why,  in  this  study,  a 
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screen  capture  was  provided  for  the  controllers  with  the  questionnaires  to  improve  their 
ability  to  recall  the  scenario. 

For  the  third  hypothesis,  self-reported  questions  about  distraction  were  asked  of 
the  participants  during  each  of  the  post-Questionnaires,  and  the  answers  were  compared 
within  participants  on  the  control  and  experimental  conditions. 

For  the  last  hypothesis,  self-reported  questions  about  perceived  difficulty  and 
facilitation  of  the  task  by  the  use  of  the  text  on-screen  were  asked  of  the  participants 
during  the  post-questionnaire.  The  answers  were  again  compared  within  participants  on 
the  control  and  experimental  conditions.  A  summary  of  the  hypotheses  and  measures 
used  for  this  research  is  provided  in  Table  2. 

Table  2:  Research  Hypotheses  and  Measurements 

Hypothesis  Measures  Used 


HI: 

Supplementary  text  transcription  of  pilots’  communication  will  increase  controllers’ 
comprehension  of  pilots’  transmissions 

Mouse  Clicks 

H2: 

Supplementary  text-cueing  will  improve  memory  recall  on  pilots’  transmissions 

Post-questionnaire  - 
Query  technique 

H3: 

Supplemental  text  box  will  increase  controller's  distraction 

Post  questionnaire  - 
Self-reported 

H4: 

Additional  text-cueing  will  reduce  perceived  difficulty  of  the  communication  task 
between  controllers  and  pilots 

Post  questionnaire  - 
Self-reported 

Pilot  Study 

A  pilot  study  was  carried  out  approximately  three  weeks  before  the  initial 
experiment.  The  pilot  study  was  conducted  with  6  volunteers,  from  which  2  were  Air 
Force  Institute  of  Technology  graduate  students,  3  were  air  traffic  controllers  from 
Wright  Patterson  AFB  Tower  and  1  was  a  Human  Factors  researcher  and  a  former  airline 
pilot  in  a  campus  room  on  WPAFB,  Dayton,  OH. 
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The  pilot  study  was  conducted  with  a  control  and  experimental  scenario.  There 
was  no  training  section.  The  experimental  scenario  included  2  text  boxes  to  transcribe 
audio  communications  from  pilots. 

The  goals  of  the  pilot  study  were:  (1)  check  the  validity  of  the  current  experiment 
structure,  (2)  check  the  validity  of  the  current  questionnaire  structure,  (3)  gather  feedback 
on  the  test  and  survey,  and  (4)  verify  the  proposed  technology. 

Useful  remarks  were  received  from  the  pilot  study.  The  initial  use  of  two  text 
boxes  was  viewed  as  distracting  due  to  the  lack  of  training  for  the  new  technology  and  no 
usable  data  was  collect.  Since  the  new  technology  was  not  used  by  participants,  a  training 
session  was  included  to  familiarize  the  participants  with  the  experiment  flow  and 
purpose.  The  text  box  was  adjusted  in  accordance  with  notable  remarks  to  make  it  more 
readable  and  useful.  Before  the  pilot  study,  the  text  transcription  was  all  white,  but  after 
some  controllers’  remarks,  the  call-signs  of  the  aircraft  were  coded  to  yellow  in  the  text- 
transcriptions  to  help  the  controller  maintain  productive  scanning  by  facilitating  the 
identification  of  the  aircraft  that  made  the  call. 

Summary 

This  chapter  presented  the  methodology  for  procuring  data  correlating  theory  to 
controller’s  efficiency.  Specifically  a  research  design  was  operationalized  to  an 
experimental  model  which  could  quantitatively  measure  the  proposed  hypothesis. 
Treatments  and  questionnaires  were  administered  to  participants  on  control  and 
experimental  conditions  so  measurements  could  be  obtained  to  enable  a  comparative 
analysis  within  participants.  Finally,  this  chapter  concluded  with  a  complete  description 
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of  the  procedures  used  in  the  experiment  and  instrument  measurements  and  reliability, 
which  lends  further  credence  to  the  plausibility  of  data  to  be  analyzed  in  the  following 
chapter. 
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IV.  Analysis  and  Results 


Chapter  Overview 

The  previous  chapter  identified  the  methodology  for  acquiring  data  in  order  to 
support  the  hypotheses.  This  chapter  presents  an  analysis  of  data  collected  during  the 
experiment  and  interprets  results.  Each  hypothesis  proposed  on  Chapter  2  is  analyzed.  All 
the  statistical  tests  in  this  study  were  conducted  using  SPSS  software. 

The  independent  t-test  was  conducted  to  analyze  if  there  is  statistically  significant 
difference  between  the  means  in  the  control  and  the  experimental  groups. 

In  the  case  that  the  t-test  was  significant,  stepwise  regression  was  conducted  to 
identify  significance  between  control  and  experimental  groups  for  two  reasons.  First, 
stepwise  regression  it  is  the  fastest  way  to  analyze  the  data  when  there  are  many 
variables,  especially  when  using  a  tool  as  advanced  as  SPSS  or  similar  statistical  software 
package.  Second,  in  this  procedure,  each  time  a  new  variable  is  added  to  the  model,  the 
significance  of  each  of  the  variables  already  in  the  model  is  re-examined,  and  this  process 
continues  until  no  more  variables  can  be  added  or  removed,  leaving  only  the  significant 
variables  in  the  model. 

Results 

Realism  was  measured  to  validate  the  scenarios  and  the  use  of  the  new 
technology.  There  was  no  statistically  significant  difference  between  groups,  which 
means  that  both  scenarios  were  perceived  to  be  equally  realistic.  Also,  94.3%  (33/35)  of 
the  participants  reported  that  they  found  the  technology  to  be  useful  in  this  capacity. 
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The  dependent  variables  for  this  study  were  number  of  clicks  (HI),  recall  (H2), 
distraction  (H3)  and  difficulty  (H4).  The  main  independent  variable  was  group  (control 
and  experimental).  Other  independent  variables  also  considered  for  this  study  were 
gender,  experience,  education,  use  of  glasses,  eye  surgery,  distraction,  ability  to 
multitask,  individual  ability  to  read,  individual  ability  to  understand  transmissions,  trust, 
computer  use,  open  mind,  and  familiarity. 

All  these  independent  variables  were  considered  to  analyze  the  potential  influence 
of  those  factors  on  controllers’  performance.  The  main  concern  of  this  study  is  the 
difference  between  the  control  and  experimental  groups,  but  the  other  variables  were  also 
considered  for  their  potential  influence  on  the  results.  Gender  was  considered  to  analyze 
potential  differences  between  men  and  women,  but  no  significant  influence  was  found. 
Experience  in  ATC  was  considered  as  a  potential  variable  that  could  significantly 
influence  performance.  Education  was  considered  since  it  could  have  been  a  potential 
factor  to  understand  the  benefits  generated  by  the  use  of  new  technology.  Use  of  glasses 
and  eye  surgery  were  considered  to  study  their  potential  influence  on  participants’ 
perfonnance  while  scanning  the  radar  display  and/or  reading  text  on  screen.  Self¬ 
distraction  was  considered  to  identify  the  participants  that  are  easily  distracted  while 
working  ATC.  Individual  ability  to  multitask  was  considered  to  identify  the  participants 
that  have  trouble  multitasking,  which  could  influence  their  ability  to  listen  the 
transmission  and  read  the  on-screen  text  at  the  same  time.  Individual  ability  to  read  was 
considered  because  of  its  potential  impact  on  participants’  ability  to  rapidly  read  the  text 
on-screen.  Individual  ability  to  understand  transmissions  was  considered  to  measure 
individuals’  potential  to  comprehend  “step-ons”  during  pilots’  transmissions.  Trust  was 
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considered  to  evaluate  its  influence  on  the  use  and  individual  perfonnance  with  the 
additional  text.  Computer  use  was  considered  to  measure  time  spend  using  the  computer 
and  its  influence  on  participants’  perfonnance.  Open  mind  was  considered  to  evaluate 
participants’  receptivity  to  new  technologies  and  changes  in  their  regular  workstation. 
Familiarity  with  the  proposed  technology  was  also  considered  to  study  its  influence  on 
participants’  performance. 

An  independent  t-test  was  used  to  detennine  statistical  differences  between 
experimental  and  control  means  for  each  hypothesis.  In  the  cases  that  t-test  showed 
significance,  stepwise  regression  was  conducted  to  determine  the  influence  of  the 
independent  variables  considered  in  the  model. 

Investigative  Questions  Answered 

HI:  Supplementary  text  transcription  of  pilots’  communication  will  increase 
controllers  ’  comprehension  of  pilots  ’  transmissions 
Hypothesis  1  assumes  a  negative  relationship  between  numbers  of  clicks  and 
comprehension.  The  dependent  variable  was  number  of  clicks  (e.g.  comprehension).  High 
number  of  clicks  was  interpreted  to  imply  low  comprehension.  The  independent  variable 
was  group  (control  group  as  number  0  and  experimental  group  as  number  1)  and  other 
independent  variables  were  also  considered  for  stepwise  regression  (gender,  experience, 
education,  use  of  glasses,  eye  surgery,  self-distraction,  ability  to  multitask,  individual 
ability  to  read,  individual  ability  to  understand  transmissions,  trust,  computer  use,  open 
mind,  and  familiarity).  All  the  independent  variables  collected  with  the  Pre-questionnaire 
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were  considered  to  analyze  the  potential  influence  of  those  factors  on  controllers’ 
comprehension. 

An  independent  t-test  was  conducted  with  95%  Confidence  Interval  (Cl)  to  check 
for  significant  difference  between  control  and  experimental  groups,  see  Tables  3  and  4, 
and  since  significance  was  found,  stepwise  regression  was  conducted  to  analyze  the 
influence  of  the  independent  variables  on  comprehension,  see  Tables  5  and  6. 

The  data  was  assumed  normally  distributed  for  both  groups,  since  a  t-test  is  a 
robust  test  with  respect  to  the  assumption  of  normality,  and  there  was  homogeneity  of 
variance  as  assessed  by  Levene’s  Test  for  Equality  of  Variances.  Therefore,  an 
independent  t-test  was  run  on  the  data  as  well  as  95%  Cl  for  the  mean  difference.  It  was 
found  that  number  of  clicks  in  the  experimental  group  (group  1,  1.14  +/-  2.341)  were 
significantly  lower  than  for  the  control  group  (group  0,  2.34  +/-  1.327)  at  t(68)  =  2.638,  p 
=  .010  with  a  difference  of  1.2  (95%  Cl,  .292  to  2.108). 


Table  3:  Descriptive  Statistics  HI 


Group 

Mean 

Std. 

Deviation 

N 

0 

2.34 

1.327 

35 

1 

1.14 

2.341 

35 

Total 

1.74 

1.983 

70 

Table  4:  Independent  Samples  Test  -  HI 


Levene's  Test  for 

Equality  of 
Variances 

t-test  for  Equality  of  Means 

Sig. 

(2- 

Mean 

Std.  Error 

95%  Cl  of  the 
Difference 

F 

Sig. 

T 

Df 

tailed) 

Difference 

Difference 

Lower 

Upper 

Clicks 

Equal 

1.653 

.203 

2.638 

68 

.010 

1.200 

.455 

.292 

2.108 

variances 

assumed 

Equal 

variances 

not 

assumed 

2.638 

53.813 

.011 

1.200 

.455 

.288 

2.112 
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A  statistically  significant  difference  ( p  <  .05)  was  discovered  between  the 
treatment  and  control  groups,  identifying  a  definitive  change  in  comprehension. 
Therefore,  stepwise  regression  was  run  to  detennine  the  influence  of  the  independent 
variables  on  comprehension.  The  model  indicated  a  variance  explained  of  47%  (Adjusted 
R2  =  .477)  with  other  independent  variables  that  helped  explain  the  model,  such  as  open 
mind,  familiarity,  multitask  and  experience  at  F( 69)  =  1 3.609.  p  =  .000. 


Table  5:  Coefficients  HI 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

B 

Std.  Error 

Beta 

t 

Sig. 

5  (Constant) 

10.137 

1.456 

6.964 

.000 

Open  Mind 

-.845 

.238 

-.330 

-3.557 

.001 

Familiarity 

-.189 

.094 

-.200 

-2.014 

.048 

Group 

-1.200 

.343 

-.305 

-3.502 

.001 

Multitask 

-.358 

.128 

-.253 

-2.799 

.007 

Exper. 

(years) 

.061 

.025 

.237 

2.442 

.017 

Table  6:  Model  Summary  HI 


Model 

R 

R  Square 

Adjusted 

R  Square 

Std. 

Error  of 
the 

Estimate 

.718 

.515 

All 

1.434 

Participants  with  open  mind  that  reported  they  believe  that  new  technologies  can 
improve  ATC  had  lower  numbers  of  clicks  and  better  comprehension.  Participants  that 
reported  they  are  familiar  with  the  text  on-screen  technology  had  a  lower  number  of 
clicks  as  expected.  Participants  that  reported  they  are  able  to  multitask  and  feel 
comfortable  listening  and  reading  at  the  same  time  had  a  lower  number  of  clicks. 
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Participants  with  higher  experience  in  ATC  had  higher  numbers  of  clicks.  Individuals 
with  more  experience  may  have  been  more  resistant  to  the  change  in  presentation  of  the 
data  due  to  it  significant  difference  from  their  standard  workstation.  However, 
inexperienced  controllers  may  have  been  more  receptive  to  the  new  display  not  having 
been  fully  acclimated  to  the  current  standard  display  like  the  experienced  group. 

The  findings  for  Hypothesis  1  indicate  a  strong  relationship  between  the  interface 
and  the  number  of  clicks  which  indicates  that  text  cueing  improved  comprehension.  The 
number  of  mouse  clicks  for  the  treatment  group  was  significantly  lower  than  for  the 
control  group,  implying  that  the  experimental  group’s  comprehension  was  improved 
through  the  availability  of  text  cueing. 

H2:  Supplementary  text-cueing  will  improve  memory  recall  on  pilots’ 

transmissions 

Hypothesis  2  predicts  an  improvement  in  memory  recall  with  the  use  of  the  text 
on-screen.  The  dependent  variable  was  the  number  of  correct  answers  for  the  recall 
questions.  The  independent  variable  was  group  (control  group  as  number  0  and 
experimental  group  as  number  1). 

Independent  t-test  was  conducted  with  95%  Confidence  Interval  (Cl)  to  check  for 
significant  difference  between  control  and  experimental  groups,  see  Tables  7  and  8. 

The  data  was  assumed  normally  distributed  for  both  groups,  since  t-test  is  a  robust 
test  with  respect  to  the  assumption  of  normality,  and  there  was  homogeneity  of  variance 
as  assessed  by  Levene’s  Test  for  Equality  of  Variances.  Therefore,  an  independent  t-test 
was  run  on  the  data  as  well  as  95%  Cl  for  the  mean  difference.  It  was  found  recall  in  the 
experimental  group  (group  1,3.14+/-  1.517)  was  not  significantly  different  than  the 
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control  group  (group  0,  3.2  +/-  1.208)  at  t{ 68)  =  0.174,p  =  .862  with  a  difference  of  0.057 


(95%  Cl,  -.597  to  .711). 


Table  7:  Descriptive  Statistics  H2 


Group 

Mean 

Std. 

Deviation 

N 

0 

3.20 

1.208 

35 

1 

3.14 

1.517 

35 

Total 

3.17 

1.362 

70 

Table  8:  Independent  Samples  Test  -  H2 


Levene's  Test  for 

Equality  of  Variances 

t-test  for  Equality  of  Means 

Sig. 

(2- 

Mean 

Std.  Error 

95%  Cl  of  the 
Difference 

F 

Sig. 

T 

Df 

tailed) 

Difference 

Difference 

Lower 

Upper 

Recall  Equal 

1.759 

.189 

.174 

68 

.862 

.057 

.328 

-.597 

.711 

variances 

assumed 

Equal 

variances 

not 

assumed 

.174 

64.743 

.862 

.057 

.328 

-.598 

.712 

No  statistically  significant  difference  ( p  >  .05)  was  discovered  between  the 
treatment  and  control  groups.  The  findings  for  Hypothesis  2  indicate  that  there  is  no 
significant  difference  in  memory  recall  due  to  the  use  of  the  new  technology. 

The  role  of  memory  has  been  heavily  researched  in  the  air  traffic  environment  and 
its  importance  concerning  whether  the  controllers  should  retain  memory  or  not  has  been 
debated.  Some  studies  defend  that  controllers  deal  with  a  constantly  dynamic 
environment  and  therefore  deal  with  dynamic  memory  since  the  scene  changes  every 
time.  Hopkin  (1980)  stated  that  the  ability  to  forget  the  information  is  as  important  as  the 
ability  to  remember,  allowing  them  to  deal  with  just  the  necessary  infonnation  required 
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and  avoid  the  workload  associated  with  keeping  memory.  A  solution  to  the  limitations 
and  constraints  of  memory,  according  to  Stein  and  Garland  (1993),  is  to  provide  a  more 
reliable  visual  display  that  shall  increase  the  transient  visual  and/or  auditory  infonnation. 
According  to  the  authors: 

“Such  memory  aiding  would  allow  prolongation  and  subsequent  storing  of  the 
information,  potentially  enhancing  the  fidelity  of  working  memory.  A  controller 
would  then  not  need  to  overcome  the  fragilities  of  working  memory,  because  the 
information  would  be  physically  present,  requiring  only  simple  perceptual 
processing.” 

This  additional  information  mentioned  by  the  authors  is  also  provided  by  the 
supplemental  text  that  will  be  available  for  controllers  on  screen  as  a  memory  aid. 

Even  if  doesn’t  improve  memory  recall,  the  text  on  screen  still  provide  a  good  aid 
because  controller  can  have  the  ability  to  quickly  find  and  look  the  infonnation  instead  of 
relying  on  his/her  memory. 

H3:  Supplemental  text  box  will  increase  controller's  distraction 
Hypothesis  3  predicts  an  increasing  distraction  with  the  use  of  the  text  on-screen. 
The  dependent  variable  was  distraction,  self-reported  by  controllers  in  the  post¬ 
questionnaires.  The  independent  variables  were  group  (control  group  as  number  0  and 
experimental  group  as  number  1)  and  other  independent  variables  were  also  considered 
for  stepwise  regression  (gender,  experience,  education,  use  of  glasses,  eye  surgery,  self¬ 
distraction,  ability  to  multitask,  individual  ability  to  read,  individual  ability  to  understand 
transmissions,  trust,  computer  use,  open  mind,  and  familiarity).  All  the  independent 
variables  collected  with  the  Pre-questionnaire  were  considered  to  analyze  the  potential 
influence  of  those  factors  on  controllers’  distraction. 
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Independent  t-test  was  conducted  with  95%  Confidence  Interval  (Cl)  to  check  for 
significant  difference  between  control  and  experimental  groups,  see  Tables  9  and  10,  and 
then  stepwise  regression  was  conducted  in  order  to  analyze  the  influence  of  the 
independent  variables  on  distraction,  see  Tables  1 1  and  12. 

The  data  was  assumed  normally  distributed  for  both  groups,  since  t-test  is  a  robust 
test  with  respect  to  the  assumption  of  normality,  and  there  was  homogeneity  of  variance 
as  assessed  by  Levene’s  Test  for  Equality  of  Variances.  Therefore,  an  independent  t-test 
was  run  on  the  data  as  well  as  95%  Cl  for  the  mean  difference.  It  was  found  that 
distraction  in  the  experimental  group  (group  1,  6.09  +/-  3.381)  was  significantly  higher 
than  the  control  group  (group  0,  4.57  + 1-2.119 )  at  /(68)  =  -2.047,  p  =  .045  with  a 
difference  of-1.514  (95%  Cl,  -2.991  to  -.038). 


Table  9:  Descriptive  statistics  H3 


Group 

Mean 

Std. 

Deviation 

N 

0 

4.57 

2.779 

35 

1 

6.09 

3.381 

35 

Total 

5.33 

3.166 

70 

Table  10:  Independent  Samples  Test  -  H3 


Levene's 
Test  for 
Equality  of 

Variances 

t-test  for  Equality  of  Means 

Sig.  (2- 

Mean 

Std. 

Error 

Differen 

95%  Cl  of  the 
Difference 

F 

Sig. 

t 

Df 

tailed) 

Difference 

ce 

Lower 

Upper 

Distraction  Equal 

2.623 

.110 

-2.047 

68 

.045 

-1.514 

.740 

-2.991 

-.038 

variances 

assumed 

Equal 

variances 

not 

assumed 

-2.047 

65.540 

.045 

-1.514 

.740 

-2.992 

-.037 
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A  statistically  significant  difference  ( p  <  .05)  was  discovered  between  the 
treatment  and  control  groups,  but  the  model  only  explained  9.3%  of  the  variance 
(Adjusted  R2  =  .093).  Due  to  the  very  low  variance  explained,  definitive  conclusions 
cannot  be  drawn  for  Hypothesis  3  at  F{ 69)  =  4.537,  p  =  .014. 


Table  11:  Coefficients  Table  H3  -  Model 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

T 

Sig. 

B 

Std. 

Error 

Beta 

2  (Constant) 

9.114 

2.166 

4.209 

.000 

Multitask 

-.560 

.259 

-.247 

2.158 

.034 

Group 

1.514 

.721 

.241 

2.101 

.039 

a.  Dependent  Variable:  Distraction 


Table  12:  Model  Summary  H3 


Model 

R 

R  Square 

Adjusted 

R  Square 

Std.  Error  of 
the  Estimate 

2 

.345 

.119 

.093 

3.015 

b.  Predictors:  (Constant).Multitask,  Group 


Participants  that  reported  they  are  able  to  multitask  and  feel  comfortable  listening 
and  reading  at  the  same  time  had  lower  distraction  caused  by  the  additional  text  on 
screen. 

The  findings  for  Hypothesis  3  indicate  a  significant  difference  between  the 
treatment  and  control  groups  on  distraction.  Additionally,  participants  that  self-reported 
high  ability  to  multitask  and  feel  comfortable  reading  and  listening  at  the  same  time  had 
lower  distraction.  The  distraction  for  the  treatment  group  was  significantly  higher  than 
for  the  control  group,  but  no  definitive  conclusions  can  be  made  due  to  the  very  low 
variance  explained. 
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H4:  Additional  text-cueing  will  reduce  perceived  difficulty  of  the 


communication  task  between  controllers  and  pilots 

Hypothesis  4  predicts  a  decrease  in  the  perceived  difficulty  of  the  task  with  the 
use  of  the  text  on-screen.  The  dependent  variable  was  difficulty,  self-reported  by 
controllers  in  the  post-questionnaires.  The  independent  variable  was  (control  group  as 
number  0  and  experimental  group  as  number  1). 

Independent  t-test  was  conducted  with  95%  Confidence  Interval  (Cl)  to  check  for 
significant  difference  between  control  and  experimental  groups,  see  Tables  13  and  14. 

The  data  was  assumed  normally  distributed  for  both  groups,  since  t-test  is  a  robust 
test  with  respect  to  the  assumption  of  normality,  and  there  was  homogeneity  of  variance 
as  assessed  by  Levene’s  Test  for  Equality  of  Variances.  Therefore,  an  independent  t-test 
was  run  on  the  data  as  well  as  95%  Cl  for  the  mean  difference.  It  was  found  difficulty  in 
the  experimental  group  (group  1,  8.17  +/-  2.065)  was  not  significantly  different  than  the 
control  group  (group  0,  7.40  +/-  1.802)  at  t{ 68)  =  -1.665,  p  =  .100  with  a  difference  of  - 
.771  (95%  Cl, -1.696  to.  153). 


Table  13:  Descriptive  Statistics  H4 


Group 

Mean 

Std. 

Deviation 

N 

0 

7.40 

1.802 

35 

1 

8.17 

2.065 

35 

Total 

7.79 

1.963 

70 
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Table  14:  Independent  Samples  Test  -  H4 


Levene's 
Test  for 

Equality  of 
Variances 

t-test  for  Equality  of  Means 

95% 

Sig. 

(2- 

Mean 

Std.  Error 

Confidence 
Interval  of  the 
Difference 

F 

Sig. 

t 

Df 

tailed) 

Difference 

Difference 

Lower 

Upper 

Difficulty  Equal 

.005 

.941 

-1.665 

68 

.100 

-.771 

.463 

-1.696 

.153 

variances 

assumed 

Equal 

variances 

not 

assumed 

-1.665 

66.776 

.101 

-.771 

.463 

-1.696 

.153 

No  statistically  significant  difference  ( p  >  .05)  was  discovered  between  the 
treatment  and  control  groups.  The  findings  for  Hypothesis  4  indicate  that  there  is  no 
significant  difference  in  perceived  difficulty  of  the  task  due  to  the  use  of  the  new 
technology. 


Summary 

The  findings  from  this  data  analysis  identify  some  significant  findings  relative  to 
the  proposed  technology.  Specifically,  Hypothesis  1  is  supported  since  the  new 
technology  increased  participants’  comprehension.  Hypothesis  2  was  not  supported,  but 
the  text  available  on  screen  is  still  good  since  it  can  be  used  by  the  controllers  as  a 
memory  aid.  Hypothesis  3  was  not  supported,  since  any  definitive  conclusions  could  be 
drawn  due  to  the  very  low  variance  explained.  Hypothesis  4  was  also  not  supported,  as 
the  technology  did  not  affect  the  perceived  difficulty  of  the  task,  which  means  that  the 
task  was  not  perceived  more  difficulty  with  the  aid  of  the  new  technology.  The  following 
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chapter  will  discuss  and  interpret  the  results  of  this  study  and  their  impact  to  academic 
and  ATC  field. 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

The  problem  of  providing  effective  tools  for  controllers  has  long  been  discussed. 
Air  traffic  controllers  play  a  critical  role  in  the  ATCS.  The  literature  review  highlighted 
the  importance  of  considering  the  human  aspects  of  developing  decision  aids  to  enhance 
ATC  efficiency.  The  development  of  computer  generated  cueing  to  enhance  efficiency 
needs  to  be  carried  out  with  careful  study  as  the  tool  could  impact  safety  within  the 
ATCS. 

Systems  and  computer  cueing  need  to  be  designed  with  a  cognitive  engineering 
process  to  address  human  impact.  It  is  also  important  to  understand  the  limits  of 
automation  to  enhance  decision  making.  It  is  desired  that  controllers  actively  monitor  the 
situation  to  develop  a  better  situation  awareness  that  allows  understanding  of  the 
environment  and  improve  the  decision  making  process.  Given  the  critical  role  of  conflict 
detection  and  resolution  on  controllers’  decision  making,  it  is  desired  that  decision  aids 
support  this  activity. 

Guided  by  MRT,  the  real-time  application  of  supplementary  computer  cueing  to 
enhance  efficiency  needs  to  be  careful  studied.  The  competition  for  controllers’ 
attentional  resources  resulting  from  the  inclusion  of  the  text  on-screen  shall  be  carefully 
considered  to  fit  with  situational  priority.  The  supplementary  cueing  shall  provide 
decision  aids  without  reducing  the  controllers’  mental  picture  about  the  environment. 

The  focus  of  this  research  was  on  identifying  the  impacts  of  additional  on-screen 
text  cueing  on  controllers’  performance.  Specifically  understanding  the  potential  benefits 
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of  the  technology  and  if  it  should  enhance  controller  efficiency  in  a  congested 
communication  environment  is  the  research  question  addressed. 

Based  on  literature  review,  five  hypotheses  were  developed  in  Chapter  2.  A 
methodology  and  experiment  for  testing  the  hypotheses  was  presented  in  Chapter  3  and 
corresponding  data  analysis  conducted  on  the  results  in  Chapter  4.  A  summary  of  the 
findings  of  this  research  effort  is  provided  in  Table  15.  This  chapter  further  discusses  the 
findings  of  this  study,  their  contribution  to  academic  research,  their  significance  to 
practice  and  relevance  to  the  Air  Force,  limitations  of  this  study,  and  recommendations 
for  future  study. 


Findings 

This  study  presented  many  important  findings  that  would  be  of  great  value  to  the 
future  application  of  speech-to-text  technology  in  an  ATC  environment,  as  shown  in 
Table  15. 


Table  15:  Summary  of  Findings 

Hypothesis  Finding 


HI: 

Supplementary  text  transcription  of  pilots’  communication  will  increase  controllers’ 
comprehension  of  pilots’  transmissions 

Supported 

H2: 

Supplementary  text-cueing  will  improve  memory  recall  on  pilots’  transmissions 

Not  Supported 

H3: 

Supplemental  text  box  will  increase  controller's  distraction 

Not  Supported 

H4: 

Additional  text-cueing  will  reduce  perceived  difficulty  of  the  communication  task  between 
controllers  and  pilots 

Not  Supported 

As  predicted  in  HI,  the  additional  text  cueing  had  a  significant  effect  on 
controllers’  comprehension  of  pilots’  transmissions.  The  number  of  transmissions  that 
controllers  would  need  to  emit  to  be  able  to  answer  pilots  was  significant  lower  with  the 
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use  of  the  additional  text  on-screen.  This  fact  is  of  great  importance  since  the  reduced 
number  of  transmissions  would  potentially  reduce  frequency  congestion,  which  is  a  very 
sensitive  situation  that  can  potentially  harm  safety.  Another  important  contribution  of  the 
reduced  number  of  transmissions  is  that  it  automatically  reduces  time  needed  to  answer 
each  pilot's  transmission,  allowing  controllers  to  continually  focus  their  attention  on 
monitoring  the  airspace  instead  of  trying  to  comprehend  previous  transmissions. 

Additionally,  as  expected,  familiarity  had  a  significance  influence  on  controllers’ 
comprehension.  This  finding  is  consistent  with  the  Channel  Expansion  Theory  (Carlson 
and  Zmud,  1994)  that  states  the  richness  of  the  medium  is  not  only  a  characteristic  of  the 
channel  itself  but  is  also  influenced  by  the  perceptions  of  the  user  based  on  familiarity 
and  experience  with  the  medium.  This  fact  is  important  in  order  to  determine  that  future 
training  and  increasing  familiarity  may  also  increase  the  richness  of  the  technology  and 
its  potential  benefits  for  the  ATC  community. 

According  to  the  results  for  H2,  H3  and  H4,  the  proposed  technology  had  no 
influence  on  memory  recall  of  pilot's  transmissions,  perceived  difficulty  of  the  task  or 
distraction.  The  fact  that  the  proposed  technology  did  not  affect  perceived  difficulty  of 
the  communication  task  is  a  very  important  and  interesting  finding  due  to  the  fact  that  the 
additional  text  on  the  radar  screen  did  not  harm  controller's  ability  to  perfonn  their  task. 
The  proposed  text  cueing  did  not  reduce  the  perceived  difficulty  of  the  communication 
task  between  controllers  and  pilots,  but  also  did  not  affect  the  perceived  task  difficulty  in 
a  harmful  way.  Even  with  the  additional  technology  on  the  radar  screen,  controllers  did 
not  perceive  the  task  to  be  more  difficult  as  compared  to  a  regular  ATC  workstation. 


57 


Also,  as  desired  in  the  design  of  ATCS  (Cardosi  and  Murphy,  1995),  this  design 
requires  no  unaided  recall  of  infonnation.  It  not  only  does  not  place  greater  demands  on 
memory  than  the  current  workstation,  but  also  provides  a  memory  aid  for  controllers. 
Although  the  text  cueing  did  not  increase  memory  recall  it  is  still  a  very  important 
memory  aid,  since  the  information  will  be  physically  available  on  screen,  relieving  the 
controllers  from  the  overuse  of  memory  to  retain  infonnation  that  could  be  accessed 
anytime  during  the  decision  making  process. 

Additionally,  the  proposed  technology  was  hypothesized  to  increase  distraction 
due  to  the  additional  text  on-screen.  Although  distraction  did  appear  to  increase  with  the 
on-screen  technology,  no  definitive  conclusions  could  be  drawn  due  to  the  very  low 
variance  explained  in  the  model.  This  is  an  important  finding  since  the  low  adjusted  R- 
squared  may  indicate  that  while  there  was  significantly  more  distraction  in  the 
experimental  condition,  the  distraction  may  not  have  been  due  to  the  additional  text. 
Other  factors  yet  unknown  may  have  also  played  a  role. 

Limitations 

This  study  is  based  on  controller's  efficiency  in  a  created  ATC  environment  with 
a  workstation  designed  to  attend  the  research  purpose.  Though  the  participants  were  able 
to  concentrate  and  act  like  in  a  real  ATC  workstation  and  environment,  a  more  accurate 
environment  or  a  real  training  workstation  could  be  of  great  value  when  trying  to  access 
real  measures  of  efficiency. 

Additionally,  in  this  study,  due  to  limitations  of  the  available  technology, 
controllers  were  asked  to  monitor  the  created  scenarios,  but  not  allowed  to  interact  or 
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make  changes  in  the  radar  display  or  aircraft  paths,  since  the  scenarios  were  previously 
recorded  and  presented  as  a  movie  for  the  participants  to  monitor  and  keep  track  of  the 
situation.  This  led  the  participants  to  a  more  passive  position,  although  they  were  asked  to 
active  monitor  as  they  were  to  issue  instructions  and  act  in  real-live  scenarios.  A  study 
that  allows  the  controllers  interact  with  the  scenario  would  be  a  more  realistic  approach  in 
order  to  get  measurements  of  efficiency.  Also,  this  would  allow  the  researchers  to  obtain 
objective  measurements  of  performance  such  as  time  to  respond  and  number  of 
transmissions  issued,  for  example,  instead  of  just  self-reported  measurements. 

Another  important  consideration  for  the  proposed  technology  is  that  many 
advances  have  been  done  to  increase  the  reliability  of  the  speech-to-text  transcription.  In 
this  study,  we  assumed  a  perfect  reliable  text  transcription  of  pilots'  transmissions.  An 
important  study  with  a  variation  on  the  reliability  of  the  transcriptions  would  importantly 
measure  the  impact  of  the  actual  state  of  the  technology  on  controller's  efficiency. 

As  reported  by  the  participants,  the  lack  of  training  with  the  technology  in  this 
specific  application  constituted  a  barrier  for  the  participants  to  use  the  proposed 
technology  in  its  full  capacity  during  all  their  performance  in  that  task  due  to  the  learning 
curve  spent  to  leam  how  the  technology  worked  in  this  application. 

Contributions  to  Research 

This  study  has  implications  for  future  study  and  development  of  systems  and  tools 
for  the  increasingly  congested  environment  of  ATC,  especially  when  it  comes  to 
controllers.  The  results  of  this  research  further  knowledge  on  the  impact  of  speech-to-text 
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technology  in  an  ATC  environment,  and  demonstrated  proof-of-concept  for  text-cueing 
effectiveness  under  realistic  ATC  conditions. 

This  study  is  novel  to  academic  research  in  that  it  focuses  on  the  application  of 
the  speech-to-text  technology  to  improve  controller'  efficiency  during  the  communication 
task  in  a  simulated  congested  air  traffic  environment.  The  impact  of  the  technology  on 
perfonnance  was  measured  with  actual  air  traffic  controllers,  and  their  perceptions  about 
the  usefulness  and  impacts  of  the  technology  were  reported. 

Implications  for  ATC  community  and  for  Air  Force 

The  use  of  the  proposed  technology  has  many  implications  for  ATC  community 
and  Air  Force.  The  possible  implications  of  this  study  to  the  Air  Force  are  no  different 
than  those  of  the  ATC  community  in  general.  It  suits  the  entire  community  as  the 
technology  would  apply  in  the  same  manner.  The  benefits  generated  with  the  use  of 
additional  text  may  potentially  help  improve  aerospace  safety  and  ATC  efficiency.  This 
research  represents  the  initial  steps  for  the  implementation  of  the  speech-to-text 
technology  to  improve  air  traffic  controllers'  efficiency  in  a  congested  communication 
task. 

The  primary  consequence  noticed  from  the  results  of  this  study  is  the  frequency 
decongestion  resulted  from  the  increased  comprehension  of  pilots’  transmissions.  Since 
controllers  will  be  able  to  comprehend  the  transmissions  without  the  need  of  a  “SAY 
AGAIN”  transmission,  but  will  be  able  to  indeed  read  what  was  said;  radio  congestion 
will  be  significantly  reduced  and  allow  for  a  safe  increase  in  ATC  traffic.  In  a  real-time 
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application,  this  technology  will  potentially  allow  air  traffic  controllers  to  monitor  more 
aircraft  and  large  geographical  spaces  due  to  increased  comprehension  and  efficiency. 

Additionally,  in  the  eventuality  of  accidents  or  incidents,  controllers  are  called  to 
transcribe  all  the  radio  communications  recorded  in  order  to  help  the  investigations 
regarding  the  incident.  The  ATC  recordings  are  usually  very  long  and  demands  several 
working  hours  to  be  transcribed.  In  many  cases,  there  are  also  many  unintelligible  parts 
that  can  not  be  transcribed  and  are  considered  missing  or  unintelligible.  With  the  use  of 
speech-to-text  transcription  technology,  these  transcriptions  will  be  automatically  done  in 
real  time,  saving  time  and  personnel  potentially  allocated  to  these  tasks.  The  ATC 
recordings  will  be  already  transcribed  and  the  availability  of  persistent  transcriptions  of 
all  radio  communication  will  be  automatically  provided  by  the  proposed  technology  for 
consultation  at  any  time  it  is  needed. 

In  accordance  to  Weinstein  (1991),  in  a  paper  issued  from  Proceeding  of  the 
IEEE:  “Training  for  military  (or  civilian)  air  traffic  controllers  (ATC’s)  represents  an 
excellent  application  for  speech  recognition  systems”.  According  to  the  author,  a  number 
of  different  studies  have  been  done  using  this  technology  to  improve  ATC  training,  such 
as:  the  potential  to  eliminate  pseudo-pilots;  prototype  ATC  trainers;  and  integration  with 
complex  training  systems  including  displays  and  scenario  creation. 

Another  potential  training  application  is  the  use  of  the  generated  text  transcription 
by  instructors  to  show  the  controller's  his/her  perfonnance  during  the  training.  The 
transcription  could  be  used  to  show  them  what  the  request  was  and  how  the  controllers 
responded  in  order  to  teach  and  show  examples  and  easily  identify  errors  associated  with 
wrong  instructions  emitted  by  controllers  while  training. 
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Besides  that,  as  reported  by  the  participants,  the  use  of  speech-to-text 
transcriptions  would  be  also  of  great  value  to  reduce  difficulty  associated  with 
understanding  foreign  accents.  This  is  an  important  application  since  the  correlation 
between  accents  and  speech  intelligibility  has  long  been  researched  (Varonis  and  Gass, 
1982;  Gass  and  Varonis,  1984;  Munro  and  Derwing,  1995),  and  according  to  Munro  and 
Derwing  (1995),  the  “effects  of  nonnative-like  pronunciations  on  intelligibility  are  far 
from  clear”. 

Future  Research  Recommendations 

This  research  represents  the  first  step  towards  the  implementation  of  speech-to- 
text  technology  for  air  traffic  controllers.  Many  studies  still  need  to  be  done  in  order  to 
guarantee  the  safe  application  of  the  proposed  technology  to  enhance  controllers' 
efficiency  in  ATM.  Further  aspects  of  HF  design  need  to  be  explored  to  ensure  a  user- 
centered  approach  for  the  application  of  this  technology. 

To  improve  the  results  for  this  study,  a  new  experiment  could  be  run  with  training 
for  the  new  technology.  As  it  was  reported  by  many  participants,  they  would  improve  and 
better  use  the  technology  in  all  its  extent  if  they  were  already  familiarized  to  work  with 
the  proposed  technology. 

Although  this  study  showed  interesting  results  from  controllers’  perceptions  of  the 
new  technology,  it  is  of  greater  value  to  pursue  a  study  to  obtain  objective  measures  of 
operators’  performance,  based  on  the  fact  that  perceptions  do  not  always  reflect  real 
perfonnance.  Real-time  interaction  would  be  of  great  value  to  be  observed  in  future 
studies  in  order  to  provide  objective  measures  of  performance,  such  as  time  and  accuracy 
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of  the  decisions.  An  important  study  would  allow  controllers  to  interact  in  order  to  collect 
objective  measurements  of  controllers’  perfonnance,  including  time  to  answer  requests 
with  and  without  the  text,  for  example. 

The  safety  impact  of  the  technology  still  requires  detailed  exploration.  Any 
additional  or  proposed  ATC  technology  would  need  to  carefully  consider  its  safety 
impact.  The  use  of  eye  tracker  to  detennine  time  spent  reading  (eyes  off  scope)  the  text 
would  be  of  great  value  to  deeply  investigate  the  impact  of  text  on  safety.  Future  studies 
would  combine  the  addition  of  text  cueing  with  eye  tracking  in  order  to  measure  the 
amount  of  time  that  controllers  spend  reading  the  text  instead  of  monitoring  the  radar 
display. 

Another  important  consideration  for  a  real-time  application  is  to  consider  the  use 
of  the  real  state  of  the  technology  and  study  the  use  of  real  time  speech  to  text 
transcription  and  the  impact  of  a  not  100%  reliable  speech-to-text  transcription.  For  the 
purpose  of  this  study,  a  100%  reliability  of  the  speech-to-text  transcription  was  assumed. 
Although  great  advancements  have  been  done  with  this  technology  in  reaching  higher 
levels  of  reliability,  it  is  still  not  a  100%  reliable.  Therefore,  studies  with  different  levels 
of  reliability  would  be  of  great  value  to  study  the  impact  of  the  application  of  the 
technology  in  a  real-time  situation. 

Further  research  of  the  speech-to-text  technology  itself  is  of  great  importance.  The 
ideal  design  of  the  text  box  -  and  also  others  techniques  and  capacities  associated  with 
speech-to-text  tools  requires  further  study.  The  MMC  Suite,  for  example,  has  many 
others  capacities  that  were  not  explored  in  this  research.  By  allowing  a  real-time 
interaction  with  the  participants,  the  technology  usability  itself  could  be  explored.  Other 
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capacities  could  be  researched  such  as  the  availability  of  the  audio  recording  associated 
with  the  text  transcribed  on  screen.  This  capacity  allows  the  controller  to  access  the 
recorded  audio  anytime,  clicking  on  the  respective  text  transcription,  even  without  a 
100%  text  transcription  reliability.  This  has  the  potential  to  help  decrease  frequency 
congestion  even  if  the  text  transcribed  was  not  completely  understood,  since  the 
controller  will  be  able  to  listen  to  the  audio  recorded  without  asking  the  pilots  to 
retransmit. 

Another  area  for  further  study  is  deeply  investigating  controllers'  workload  from 
the  additional  text.  One  variable  that  was  not  tested  in  the  experiment  was  the  controllers’ 
workload  or  what  effect  did  the  proposed  technology  had  on  controllers’  workload.  This 
may  be  significant  to  study,  since  it  is  an  important  characteristic  of  controllers’ 
efficiency  and  an  increase  on  workload  could  adversely  affect  the  operator’s  ability  to 
manage  complex  communications  tasks. 

Another  further  step  to  improve  controllers'  efficiency  is  to  study  the  impact  of 
providing  identification  of  the  spatial  location  of  the  aircrafts  while  pilots’  are 
transmitting  their  messages  through  radio.  Providing  spatial  cue  of  aircraft  location  while 
pilots'  are  still  transmitting  their  message  would  potentially  reduce  search  time  in  the 
radar  screen  done  by  controllers’  every  time  they  receive  a  call  from  a  different  aircraft. 
Controllers  are  required  to  identify  the  position  of  each  aircraft  before  they  can  respond 
for  its  transmissions.  This  results  in  a  recurrent  screen  search  every  time  a  pilot  calls  in 
order  to  identify  the  aircraft  and  be  able  to  answer  its  call.  An  additional  cue  of  its  spatial 
location  would  potentially  reduce  this  search  time  and  provide  controllers  with  more  time 
to  be  used  in  other  tasks. 
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Conclusion 


To  provide  cognitive  tools  to  enhance  controllers’  decision  making,  it  is  very 
important  to  first  understand  how  controllers  can  be  affected.  This  study  reviews  a 
significant  range  of  considerations  from  previous  research  in  order  to  provide  an 
overview  of  the  importance  of  human  factors  considerations  when  designing  a  cognitive 
tool  to  enhance  decision  making  in  ATC  environment. 

The  development  of  cognitive  engineered  tools  must  be  addressed  from  a  human 
factors  perspective  in  order  to  support  the  human  decision  maker.  This  paper  attempted  to 
provide  an  overview  of  the  cognitive  factors  that  must  be  considered  in  order  to  provide  a 
basis  for  the  development  of  cognitive  engineered  tool  to  enhance  efficiency  and  critical 
decision  making. 

By  optimizing  and  providing  all  the  infonnation  needed,  the  ATC  system  may 
benefit  with  highly  optimized  and  correctly  solutions  and  decisions  made  with  all 
necessary  infonnation  available.  Since  controllers  will  be  able  to  read  what  was  said 
instead  of  just  relying  on  auditory  memory  and  comprehension,  this  could  lead  to 
increased  efficiencies  in  the  NAS  throughout  reduced  frequency  congestion  and 
potentially  errors  reduction  associated  with  misunderstood  transmissions. 

There  has  been  significant  advancement  in  speech-to-text  technology  in  recent 
years.  The  application  of  this  technology  is  constantly  increasing,  specifically  in  military 
systems  (Weinstein,  1991).  Progress  associated  with  the  study  of  human  factors  design  to 
air  traffic  controls  has  the  potential  to  provide  a  valuable  tool  to  enhance  the  time-critical 
aspect  of  controllers  decision  making  in  an  increasingly  congested  communication 
environment. 
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This  paper  presented  encouraging  results  from  the  application  of  this  technology, 
transcribing  pilots’  transmissions  to  enhance  controllers’  efficiency.  This  research  was  an 
exploratory  study  to  determine  the  effectiveness  and  efficiency  gaining  potential  of  this 
new  technology  and  whether  further  research  is  warranted.  A  number  of  contributions 
and  implications  of  the  use  of  the  text  technology  were  described.  Applications  in 
training  and  real-time  air  traffic  are  seen  to  be  promising  for  the  proposed  technology. 
Additionally,  future  research  is  recommended  to  guarantee  the  advancement  of  the 
technology  and  its  application  to  enhance  the  ATCS  efficiency  and  safety. 
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Appendix  A:  IRB  Approval  Letter 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
WRIGHT-PATTERSON  AIR  FORCE  BASE  OHIO 

25  Jan  2012 

MEMORANDUM  FOR  LTCOL  BRENT  LANGHALS 
FROM:  Alan  R.  Heminger,  Ph.D. 

AFIT  IRB  Research  Reviewer 
2950  Hobson  Way 

Wright-Patterson  AFB,  OH  45433-7765 

SUBJECT:  Approval  for  exemption  request  from  human  experimentation 
requirements  (32  CFR219,  DoDD  3216.2  and  AFI  40-402)  for  study  titled  "The 
impact  on  performance  of  stereoscopic  dimensional  digital  radar  displays  and 
integrated  automated  visual  text  cuing  on  air  traffic  controllers”. 

1 .  Your  request  was  based  on  the  Code  of  Federal  Regulations,  title  32,  part  219,  section 
101,  paragraph  (b)  (2)  (i)  Infonnation  obtained  is  not  recorded  in  such  a  manner  that 
human  subjects  can  be  identified,  directly  or  through  identifiers  linked  to  the  subjects,  (ii) 
you  are  not  collecting  infonnation  that  could  reasonably  place  the  subjects  at  risk  of 
criminal  or  civil  liability  or  be  damaging  to  the  subjects  financial  standing,  employability, 
or  reputation,  and  (4)  Research,  involves  the  collection  or  study  of  existing  data, 
documents,  records,  pathological  specimens,  or  diagnostic  specimens,  if  these  sources  are 
publicly  available  or  if  the  infonnation  is  recorded  by  the  investigator  in  such  a  manner 
that  subjects  cannot  be  identified,  directly  or  through  identifiers  linked  to  the  subjects. 

2.  Your  study  qualifies  for  this  exemption  because  you  are  not  collecting  sensitive  data, 
which  could  reasonably  damage  the  subjects’  financial  standing,  employability,  or 
reputation.  Further,  the  demographic  data  you  are  collecting  will  be  maintained 
separately  from  experimental  data  so  that  a  given  response  will  not  be  expected  to  map  to 
a  specific  subject. 

3.  This  determination  pertains  only  to  the  Federal,  Department  of  Defense,  and  Air 
Force  regulations  that  govern  the  use  of  human  subjects  in  research.  Further,  if  a 
subject’s  future  response  reasonably  places  them  at  risk  of  criminal  or  civil  liability  or  is 
damaging  to  their  financial  standing,  employability,  or  reputation,  you  are  required  to  file 
an  adverse  event  report  with  this  office  immediately. 

ALAN  HEMINGER,  PH.D. 

AFIT  Research  Reviewer 
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Appendix  B:  Authorization  Letter  to  access  controllers 


DEPARTMENT  OF  THE  AIR  FORCE 

HEADQUARTERS  UNITED  STATES  AIR  FORCE 
WASHINGTON  DC 


7  Jan  2012 


MEMORANDUM  FOR  AFU.'ENV 

FROM:  HQHAF/A30-BA 

1480  Air  Force  Pentagon 
Washington.  D.C.20330-1480 

SUBJECT:  Accessing  Volunteer  ATC  Trainees  at  Keesler  AFB.  MS  for  AFTT  Research 

1)  TSgt  Russi  and  his  research  team  are  granted  access  to  the  USAF  Air  Traffic  Control 
technical  training  school  to  conduct  their  research  experiment  with  cooperation  of  the  school's 
commandant,  the  career  field  manager  and  any  volunteers  willing  to  participate.  It  is  understood 
that  the  research  project  is  examining  the  potential  of  stereoscopic  (3-D)  digital  radar  display 
potential  and  textual  cueing  m  congested  communications  environments  Our  tech  school 
provides  the  perfect  environment  for  this  stud}'  without  affecting  operational  ATC  while  also 
allowing  many  AD  controllers  and  apprentices  the  opportunity  to  participate. 

2)  This  support  is  for  the  solicitation  of  volunteers  at  the  school  and  is  not  an  endorsement  or 
expression  of  interest  in  the  specific  technologies  bemg  evaluated.  We,  as  a  highly  technical 
field,  understand  the  importance  of  research  of  technologies  with  potential  for  future  use  and 
enthusiastically  support  the  AFIT  mission  of  AF  level  research  and  development. 

3)  Please  direct  any  questions  regarding  this  correspondence  to  me  at:  ioseph  kirk<nis  af  mil. 


KIRK JOSEPH.C.  102 
8631681  ESSSTESS! 

JOSEPH  C  KIRK  CMSgt  USAF 
1C1  Career  Field  Manager 


68 


Appendix  C:  Informed  Consent 


Informed  Consent  Document 
For 

The  effect  of  stereoscopic  dimensional  digital  radar  displays  and  integrated 
automated  visual  text  cuing  on  air  traffic  controller  performance 

AFIT/ENY,  Air  Force  Institute  of  Technology,  Wright -Patterson  AFB  OH 


Principal  Investigator: 

Lt  Col  Brent  T.  Langhals,  DSN  785-3636,  ext.  4352, 

AFIT/ENV 

Brent.Langhals@afit.edu 

Associate  Investigators: 

TSgt  Jason  Russi,  DSN  785-3636,  ext.  4352,  AFIT/ENV 

Jason.Russi@afit.edu 

1  LT  Laurienne  Santana  (Brazilian  Air  Force),  DSN  785- 

3636,  ext.  4352,  AFIT/ENV 

Laurienne.Santana.br@afit.edu 

1.  Nature  and  purpose:  You  are  being  invited  to  take  part  in  a  research  study.  The 
information  in  this  fonn  is  provided  to  help  you  decide  whether  or  not  to  take  part. 
Study  personnel  will  be  available  to  answer  your  questions  and  provide  additional 
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information.  If  you  decide  to  take  part  in  the  study,  you  will  be  asked  to  sign  this 
consent  form.  A  copy  of  this  form  will  be  given  to  you.  Your  participation  will  occur 
at  Keesler  AFB,  MS. 

The  purpose  of  this  study  is  to  detennine  to  what  extent  visual  textual  cuing 
and  3-dimensional  air  traffic  representations  may  impact  a  controller’s  perfonnance 
both  in  heightened  situational  awareness  for  longer  durations  and  reduction  in  fatigue 
induced  by  multi-tasking,  distractions,  redundant  transmissions  and  reduced  mental 
image  creation.  The  intent  is  to  study  what  of  these  aspects  can  provide  benefits  to  the 
already  task  saturated  industry  by  providing  increased  human  factors  awareness. 

The  time  requirement  for  each  volunteer  subject  is  anticipated  to  be  a  total  of 
one  hour  over  one  or  two  visits  as  it  need  not  be  continuous.  It  is  expected 
approximately  40  subjects  will  be  enrolled  in  this  study.  Subjects  shall  be  able  to 
read  and  speak  English,  be  between  18  and  60  years  old,  possess  ATC  experience  or 
at  least  conceptual  understanding  and  training  with  sufficient  vision  to  perfonn 
simulated  ATC  tasks  with  a  stereoscopic  monitor. 


2.  Experimental  procedures:  If  you  decide  to  participate,  the  first  task  you  will 
complete  will  be  to  fill  out  a  short  questionnaire  to  capture  some  demographic 
information.  No  personally  identifying  infonnation  will  be  asked  of  you  in  this 
questionnaire. 


Next  we  will  ask  you  to  assume  the  role  of  an  air  traffic  controller.  You  will 
be  first  presented  with  a  common  scenario  to  start  the  air  traffic  control  activities, 
than  you  will  be  presented  with  a  series  of  simulated  air  traffic  situations  with  textual 
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cues,  and  after  that  you  will  be  exposed  to  3-D  displays.  Your  task  is  to  observe  each 
simulation  as  different  cues  and  presentations  are  added.  You  will  be  trained  on  3-D 
displays  and  have  an  overview  on  the  station  you  will  deal  with  prior  to  the  start  of 
the  experiment.  No  personally  identifying  infonnation  will  be  kept.  About  one  hour 
will  be  needed  to  complete  this  study. 

Your  participation  in  this  study  is  voluntary.  You  will  not  lose  any  benefit 
that  would  normally  be  entitled  if  you  do  not  participate  or  withdraw  from  the 
research.  You  may  decide  to  not  begin  or  to  stop  the  study  at  any  time.  If  you  are  a 
student,  your  refusal  to  participate  will  have  no  effect  on  your  student  status.  Also, 
any  new  infonnation  discovered  about  the  research  will  be  provided  to  you.  This 
information  could  affect  your  willingness  to  continue  your  participation  and  will 
therefore  be  furnished  to  you. 


3.  Discomfort  and  risks:  The  tasks  that  you  will  be  doing  have  no  known  safety  or 
psychological  risks.  Although  we  have  tried  to  avoid  risks,  you  may  feel  that  some 
questions  or  procedures  we  ask  you  to  do  may  be  stressful,  or  possibly  even  cause 
you  to  feel  fatigue.  If  this  occurs  you  can  stop  participating  immediately,  however 
this  is  what  we  are  evaluating  so  please  feel  free  to  express  these  feelings  and 
concerns  as  they  arise.  We  can  give  you  information  about  individuals  who  may  be 
able  to  help  you  with  these  problems  should  they  go  beyond  the  scope  of  the  trial. 


Additionally,  for  this  research  study  we  may  be  using  3-D  glasses  for  viewing 
the  displays.  There  are  no  known  risks  from  using  this  equipment  as  they  are 
commonly  available  commercial  models. 


71 


4.  Precautions  for  female  subjects  or  subjects  who  are  or  may  become  pregnant 
during  the  course  of  this  study:  There  are  no  precautions  for  female  subjects  or 
subjects  who  are  or  may  become  pregnant  during  the  course  of  this  study. 


5.  Benefits:  You  are  not  expected  to  benefit  directly  from  participation  in  this  research 
study. 


6.  Compensation:  If  you  are  active  duty  military  you  will  receive  your  normal  active 
duty  pay. 

7.  Alternatives:  Your  alternative  is  to  choose  not  to  participate  in  this  study.  Refusal 
to  participate  will  involve  no  penalty  or  loss  of  benefits  to  which  you  are  otherwise 
entitled.  You  may  discontinue  participation  at  any  time  without  penalty  or  loss  of 
benefits  to  which  you  are  otherwise  entitled.  Notify  any  investigator  of  this  study  to 
discontinue. 


Entitlements  and  confidentiality: 


a.  Records  of  your  participation  in  this  study  may  only  be  disclosed  according  to 
federal  law,  including  the  Federal  Privacy  Act,  5  U.S.C.  552a,  and  its 
implementing  regulations  and  the  Health  Insurance  Portability  and  Accountability 
Act  (HIPAA),  and  its  implementing  regulations,  when  applicable,  and  the 
Freedom  of  Information  Act,  5  U.S.C.  Sec  522,  and  its  implementing  regulations 
when  applicable.  Your  personal  information  will  be  stored  in  a  locked  cabinet  in 
an  office  that  is  locked  when  not  occupied.  Electronic  files  containing  your 
personal  infonnation  will  be  password  protected  and  stored  only  on  a  secure 
server.  It  is  intended  that  the  only  people  having  access  to  your  information  will 
be  the  researchers  named  above,  the  AFRL  Wright  Site  IRB  or  any  other  IRB 
involved  in  the  review  and  approval  of  this  protocol.  When  no  longer  needed  for 
research  purposes  your  information  will  be  destroyed  in  a  secure  manner 
(shredding).  Complete  confidentiality  cannot  be  promised,  in  particular  for 
military  personnel,  whose  health  or  fitness  for  duty  infonnation  may  be  required 
to  be  reported  to  appropriate  medical  or  command  authorities.  If  such 
information  is  to  be  reported,  you  will  be  informed  of  what  is  being  reported  and 
the  reason  for  the  report. 


b.  Your  entitlements  to  medical  and  dental  care  and/or  compensation  in  the  event  of 
injury  are  governed  by  federal  laws  and  regulations,  and  that  if  you  desire  further 
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information  you  may  contact  the  base  legal  office  (ASC/JA,  257-6142  for  Wright- 
Patterson  AFB). 


c.  If  an  unanticipated  event  (medical  misadventure)  occurs  during  your  participation 
in  this  study,  you  will  be  infonned.  If  you  are  not  competent  at  the  time  to 
understand  the  nature  of  the  event,  such  information  will  be  brought  to  the 
attention  of  your  next  of  kin  or  other  listed  emergency  contact. 


Next  of  kin  or  emergency  contact  information: 


Name 

Phone# 


d.  The  decision  to  participate  in  this  research  is  completely  voluntary  on  your  part. 
No  one  may  coerce  or  intimidate  you  into  participating  in  this  program.  You  are 
participating  because  you  want  to.  Lt  Col  Brent  T.  Langhals,  or  an  associate,  has 
adequately  answered  any  and  all  questions  you  have  about  this  study,  your 
participation,  and  the  procedures  involved.  Lt  Col  Brent  T.  Langhals  can  be 
reached  at  (937)  255-3636  ext  4352.  Lt  Col  Brent  T.  Langhals,  or  an  associate 
will  be  available  to  answer  any  questions  concerning  procedures  throughout  this 
study.  If  significant  new  findings  develop  during  the  course  of  this  research, 
which  may  relate  to  your  decision  to  continue  participation,  you  will  be  infonned. 
Refusal  to  participate  will  involve  no  penalty  or  loss  of  benefits  to  which  you  are 
otherwise  entitled.  You  may  discontinue  participation  at  any  time  without  penalty 
or  loss  of  benefits  to  which  you  are  otherwise  entitled.  Notify  one  of  the 
investigators  of  this  study  to  discontinue. 


e.  Personal  Identifiable  Infonnation  to  be  obtained  for  this  study  includes  gender, 
ethnicity,  country  of  citizenship,  age,  and  experience.  Signing  this  document  in 
no  way  alters  your  ability  to  obtain  medical  treatment  that  is  not  part  of  this  study, 
Any  Private  Health  Information  obtained  in  the  course  of  this  study  may  be  used 
by  the  investigator  unless  you  revoke  authorization  to  do  so  in  writing.  If  your 
data  is  disclosed  by  the  investigator  to  one  of  the  parties  listed  above,  those 
parties  may  pass  on  your  data  without  further  notification  to  you.  Data  collected 
in  the  course  of  this  study  may  be  withheld  from  you  by  the  investigator  for  the 
duration  of  the  study.  If  withheld,  your  data  will  be  released  at  the  conclusion  of 
the  study. 
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f.  Your  participation  in  this  study  will  not  be  photographed,  filmed  or 
audio/videotaped. 


YOU  ARE  MAKING  A  DECISION  WHETHER  OR  NOT  TO  PARTICIPATE. 
YOUR  SIGNATURE  INDICATES  THAT  YOU  HAVE  DECIDED  TO  PARTICIPATE 
HAVING  READ  THE  INFORMATION  PROVIDED  ABOVE. 


Volunteer 

Signature _ Date_ 

Volunteer  Name  (printed) _ 

Advising  Investigator  Signature _ Date 

Investigator  Name  (printed) _ 

Witness  Signature _  Date _ 

Witness  Name  (printed) _ 
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Appendix  D:  Timeline  Procedure 


Experiment  Design  Timeline: 

Initial  Briefing 

TIME  DESCRIPTION 

T  +0  Subjects  arrive  at  experiment  room  1,  adjacent  to  room  2  where  researcher 

2  will  initiate  scenario.  Consent  form  introduced  and  signed  by  subject.  (7 
min)  ( DISTRIBUTE  CONSENT  FORM  AND  PARTICIPANT  NUMBER) 

“I  am  TSgt  Jason  Russi  and  this  is  my  colleague,  Lt.  Santana  of  the 
Brazilian  Air  Force.  We  would  like  to  welcome  you  and  thank  you  for  your  voluntary 
participation  in  this  valuable  research.  This  study  will  take  approximately  1  %  hours  to 
complete.  Before  we  begin,  please  take  a  moment  to  read  through  the  consent  form  Lt. 
Santana  is  handing  you  now  with  your  participant  number.  ( DISTRIBUTE  CONSENT 
FORM  AND  PARTICIPANT  NUMBER ).  Keep  this  number  as  it  will  be  used  as  your 
sole  identification  throughout  the  experiment.  If  you  have  any  questions,  please  do  not 
hesitate  to  ask  either  one  of  us  at  any  point  now  or  during  the  briefings.  Of  note,  your 
responses  to  this  experimental  scenario  will  not  be  used  beyond  our  research  and  will 
be  destroyed  upon  completion  of  this  study.  Once  you  have  read  through  the  entire 
consent  form  and  agree  to  continue  to  participate  in  this  study,  please  sign  and  date  the 
consent  forms  and  hand  them  to  either  one  of  us.  ” 

Wait  for  participant  to  fill  out  consent  form.  (5  min) 

T  +7  Subjects  read  and  fill  in  pre-experiment  questionnaire.  See  Appendix  X 

for  full  text  of  Pre-Experiment  Questionnaire  (PreQ).  Questionnaire  will 
include  text  and  dimensional  display  questions  (5min)  ( DISTRIBUTE 
PreQ) 
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“Lt.  Santana  is  now  handing  out  a  questionnaire  {DISTRIBUTE  PreQ). 
As  part  of  this  study  we  need  to  collect  some  demographic  and  background 
information.  None  of  this  data  will  be  personally  identifying  and  will  in  no  way  be 
connected  to  your  name.  Please  take  a  moment  to  fill  out  this  questionnaire.  ” 

Wait  for  participant  to  fill  out  PreQ.  (5  min) 

T+  12  Subject  is  ushered  into  controlled  scenario  room/section  housing  Mock 

Control  Tenninal  (MCT).  Researcher  #1  reads  script  of  process  and  briefs 
subject  and  familiarizes  them  with  system  that  will  remain  the  same 
throughout  all  four  scenarios  (General  Training  section  of  this  doc).  Any 
questions  are  answered  to  ensure  understanding  of  performance  for 
subject.  All  scenarios  will  be  overall  briefed,  as  in  what  the  four  different 
types  consist  of  and  inform  the  subject  that  the  order  of  presentation  will 
be  random,  however,  operation  of  the  system  will  be  the  same.  The 
specific  details  of  each  scenario  (equipments  and  the  new  technology)  will 
be  introduced  right  before  each  scenario  (5  mins)  ( BRIEF  GENERAL 
SCENARIO.  JUST  BEFORE  EACH  SCENARIO  WE  WILL  BRIEF  THE 
SPECIFIC  CONDITION  TRAINING  AND  SHOW  PRINT  SCREEN 
CAPTURES  AND  EQUIPMENTS  FOR  EACH  SCENARIO) 

Training  Briefing  (2  mins) 


General 

“ You  have  been  invited  to  participate  in  this  study  due  to  your  background  or 
involvement  in  ATC;  and  you  are  expected  to  utilize  your  situation  awareness  as  you 
would  in  a  live  ATC  setting  performing  to  the  best  of  your  individual  ability. 

If  you  are  a  student  at  Keesler  AFB,  ATC  Technical  School,  this  will  in  no  way 
impact  your  training  records  or  even  be  shared  with  the  faculty  of  this  school.  There 


76 


will  be  no  adverse  actions  or  retribution  of  any  type  due  to  your  performance. 
Furthermore,  there  will  be  absolutely  no  personally  identifying  information  (PII) 
collected  or  retained  on  you  as  an  individual.  Your  participant  number  will  be  the  only 
identifying  information  recorded  for  purposes  of  this  research. 

The  purpose  of  this  study  is  to  determine  the  impact  and  effectiveness  of  the 
technologies,  NOT  the  controllers  ’  individual  performances  or  abilities. 

For  this  study  you  are  performing  duties  as  an  USAF  Air  Traffic  Controller  at 
Canyon  Airport,  an  imaginary  airfield.  You  will  be  tasked  with  observing  several 
aircraft  in  four  short  ATC  scenarios,  divided  into  two  parts.  The  first  part  will  consist 
of  2  short  scenarios  to  analyze  the  impact  of  3D  technology  for  observing  vertical 
separation;  and  the  second  part  will  consist  of  2  short  scenarios  to  analyze  the  impact 
of  additional  text  queuing  in  congested  radio  comm  environments. 

You  will  receive  a  short  briefing  about  the  specifics  of  the  technologies  right 
before  each  scenario.  During  all  the  scenarios,  you  will  be  expected  to  maintain 
situation  awareness,  allowing  us  to  evaluate  your  comprehension,  awareness  and 
decision  making.  This  performance  will  be  evaluated  on  a  series  of  measurable 
performance  characteristics  similar  to  the  standard  accepted  ATC  practices.  After  each 
scenario  you  will  be  asked  to  answer  some  questions  about  them. 

All  scenarios  will  be  viewed  in  a  darkened  room,  the  range  marks  are  set  to  10 
miles,  and  the  speed  is  twice  that  of  a  standard  simulation  to  allow  you  to  observe  more 
in  less  time.  The  time  on  screen  is  double  real  time.  Altitudes  in  data  tags  are;  green  = 
level  flight,  red  =  climbing  and  yellow  =  descending.  You  may  talk  but  there  will  be  no 
changes  to  the  scenarios.  ”  *  SEPARATE  SUBJECTS  FOR  SPECIFIC  TREATMENT 
BRIEFS 

Condition  A;  3D  Briefing 

-Equipment:  Glasses,  wireless  mouse,  ASUS  3D  monitor.  “In  the  next  2 
scenarios,  you  will  be  asked  to  view  only  the  large  ASUS  monitor.  This  mouse’s  left 
button  will  be  clicked  as  your  only  input ” 
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***(Ensure  Proper  alignment  of  viewing,  seat  height,  screen  position  etc.) 


Ac  Control  scenario  (45  sec):  “You  will  be  presented  a  unique  6  minute  air 

traffic  control  scenario.  You  are  asked  to  interact  by  clicking  the  left  mouse  button 
once  for  every  instance  you  are  unsure  two  or  more  aircraft  may  not  safely  pass  over  or 
under  one  another.  You  need  to  maintain  overall  S-A  of  the  scene  and  monitor  vertical 
separation  throughout.  Remember,  green  altitude  digits  are  level  flight,  red  is  a  climb 
and  yellow  is  descending.  All  aircraft  are  considered  IFR  in  your  area  of  jurisdiction. 
Some  may  not  be  squawking  Mode  C.  Do  not  assume  you  know  aircraft  characteristics 
based  on  callsign  or  intended  route  of  flight.  You  will  need  to  maintain  vigilance  as  if 
you  were  to  make  decisions  about  and  issue  instructions  for  the  aircraft.  After  the 
scenario,  you  will  need  to  answer  detailed  questions  about  what  you  saw,  so  please  try 
to  maintain  your  highest  level  of  situation  awareness.  Any  questions?  Remember: 

Click  when  you  think  they  may  not  safely  pass.  ”  (Answer  Q  ’s  if  posed) 

-  Ae  Experimental  scenario  (60  sec):  “You  will  be  presented  a  specific  unique  6 

V2  min  air  traffic  control  scenario.  This  scenario  will  be  in  full  stereoscopic  3-D.  In  this 
scenario,  you  will  be  asked  to  use  a  pair  of  3D  glasses  to  properly  observe  altitude 
variances.  You  will  interact  with  a  single  left  click  of  the  mouse  when  you  are  unsure 
if  aircraft  will  safely  pass  over  or  under  one  another.  *NOTE:  The  3D  disparity  view  is 
100%  reliable  and  accurately  reflects  altitude  variances.  Any  discernible  depth 
difference  indicates  at  LEAST  the  minimum  vertical  separation  of 1,000  \  You  will 
need  to  maintain  vigilance  as  if  you  were  to  make  decisions  about  and  issue 
instructions  for  the  aircraft.  After  the  scenario,  you  will  need  to  answer  detailed 
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questions  about  what  you  have  seen.  Any  questions?  ( answer  Q  ’s  if  posed)  Please  put 


on  the  glasses  now.  (HOLD  THEIR  GLASSES  UNTIL  SYNCRONIZED  AND 
ADJUST  POSTURE  AND  ALIGNMENT  TO  SCREEN).  Remember:  Click  when  you 
think  they  may  not  safely  pass  and  ANY  depth  separation  =  safe  separation!  Tell  us 
when  you  begin  seeing  3-D  ” 


Condition  II:  Text  Briefing 

-Equipment:  Headphone  (10  sec).  “In  the  next  two  scenarios  you  will  be  asked  to 
use  these  headphones  to  listen  to  pilots9  communications.  ” 

-  Bc  Control  scenario  (35  sec):  “You  will  be  presented  an  unique  3-minute  air 

traffic  control  scenario  and  you  will  hear  pilot’s  communications  to  the  controller 
without  controller  responses.  You  will  interact  with  single  left  mouse  clicks  when  you 
feel  a  “SAY  AGAIN99  transmission  is  needed  on  your  part  to  understand  a  pilot.  Please 
click  the  mouse  once  for  every  pilot  transmission  you  would  want  repeated.  They  will 
not  be  repeated,  we  are  merely  tracking  your  comprehension.  You  need  to  consider 
each  pilot’s  request  as  if  you  needed  to  reply  to  them  with  a  response  &/or  decision. 
After  the  scenario,  you  will  need  to  answer  detailed  questions  about  what  you  SAW  and 
HEARD.  Any  questions?  Please  put  your  headphones  on  now.  Remember:  Click  once 
for  every  pilot  transmission  you  would  want  repeated.  ” 

-  Be  Experimental  scenario  (45  sec):  “ You  will  be  presented  a  specific  3 -min  air 
traffic  control  simulation  of  a  congested  radio  environment  and  you  will  hear  pilot’s 
communications  to  the  controller  with  no  controller  replies.  For  this  scenario,  there 
will  be  a  narrow  text  box  at  the  right  side  of  the  radar  display  that  will  transcribe  all 
pilots’  communications  in  real-  time  ( show  screen  capture).  The  call  signs  will  show  up 
in  yellow  w/  the  most  recent  call  at  the  bottom  of  the  text  box.  This  automated 
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transcription  is  a  100%  accurate  and  reliable.  If  you  feel  that  a  “SAY  AGAIN” 
transmission  is  still  warranted,  left-click  the  mouse  once  for  every  pilot  transmission 
you  would  want  repeated.  They  will  not  be  repeated,  we  are  merely  tracking  your 
comprehension.  However,  if  you  can  determine  what  a  pilot  has  said  well  enough  to 
respond  whether  it  be  through  audio  OR  TEXT,  then  do  NOT  click  the  mouse.  Click 
the  mouse  ONLY  if  you  did  not  get  enough  information  to  understand  the  pilot.  You 
are  expected  to  utilize  the  new  technology  presented  to  the  MAXIMUM  EXTENT 
POSSIBLE  to  supplement  audio  communications  for  comprehension.  You  need  to 
consider  each  pilot’s  request  as  if  you  needed  to  reply  to  them  with  a  response  &/or 
decision.  After  the  scenario,  you  will  need  to  answer  detailed  questions  about  what  you 
SAW  and  HEARD.  Any  questions?  Please  you  can  put  your  headphones  now. 
Remember  click  once  for  every  pilot  transmission  you  would  want  repeated  if  you  feel 
that  a  “SAY AGAIN”  transmission  is  still  warranted”. 


Wait  participants  leave  the  room  if  there  is  more  than  1 

Experiment/scenarios  presentation 

T  +14  Researcher  #1  and  #2  fall  back  to  operator’s  station.(This  may  be  in  the 

same  room,  depending  upon  space  availability.  )  First  scenario  is  initiated. 
The  first  specific  training  and  scenario  is  initiated  according  to  the 
established  order  for  each  participant.  ALL  SCENARIOS  WILL  BE 
ADMINSTERED  IN  A  RANDOM  MANNER,  according  to  the 
“participants’  list”.  This  will  preclude  any  order  effects  that  may  skew 
data  collection  by  bias.  (10  mins)  (GIVE  SPECIFIC  TRAINING  AND  SET 
UP  SCENARIOl  and  MOUSE  CLICK  COUNTER ) 

T  +26  First  Subject  scenario  terminated  and  fill  in  post-experiment  questionnaire. 

Subject  briefed  to  wait  in  designated  holding  area  for  next  scenario 
preparation;  and  a  Post-  experiment  questionnaire  is  administered.  See 
Appendix  X  for  full  text  of  Post-Experiment  Questionnaire  (PoQ).  (10 
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min)  {DISTRIBUTE  PoQ  and  TRANSCRIBE  CLICK  COUNT  ONTO 


T  +36 


T  +46 


T  +56 


TOP ) 

“  ( DISTRIB  UTE  PoQ).  Please  take  a  moment  to  fill  out  this 
questionnaire  about  the  last  scenario  you  were  exposed  to.  There  is  no 
“penalty”  for  guessing  and  you  may  complete  the  questionnaire  in  any 
order  with  no  time  limit.  ”  ( offer  screen  capture) 

Subject  is  ushered  into  MCT  again.  Second  scenario  is  initiated. 
Researcher  #1  will  initiate  the  next  specific  training  for  the  second 
scenario.  The  second  scenario  is  initiated  according  to  the  established 
order  for  each  participant.  (10  min)  (GIVE  SPECIFIC  TRAINING  AND 
SET  UP  SCENARI02  ) 

Second  Subject  scenario  tenninated  and  fill  in  post-experiment 
questionnaire.  Subject  briefed  to  wait  in  designated  holding  area  for  next 
scenario  preparation;  and  a  Post-  experiment  questionnaire  is 
administered.  See  Appendix  X  for  full  text  of  Post-Experiment 
Questionnaire  (PoQ).  (10  min)  (DISTRIBUTE  PoQ  TRANSCRIBING 
CLICK  COUNT  ONTO  TOP) 

“( DISTRIBUTE  PoQ).  Please  take  a  moment  to  fdl  out  this 
questionnaire  about  the  last  scenario  you  were  exposed  to.  There  is  no 
“penalty”  for  guessing  and  you  may  complete  the  questionnaire  in  any 
order  with  no  time  limit.”  (Offer  MMC  screen  capture  if  applicable”) 
Subject  is  ushered  into  MCT  again.  Third  scenario  is  initiated.  Researcher 
#1  will  initiate  the  next  specific  training  for  the  third  scenario.  The  third 
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scenario  is  initiated  according  to  the  established  order  for  each  participant. 
(5  min)  (GIVE  SPECIFIC  TRAINING  AND  SET  UP  SCENARI03 


T  +61 


T  +66 


T  +71 


Third  Subject  scenario  terminated  and  fill  in  post-experiment 
questionnaire.  Subject  briefed  to  wait  in  designated  holding  area  for  next 
scenario  preparation;  and  a  Post-  experiment  questionnaire  is 
administered.  See  Appendix  X  for  full  text  of  Post-Experiment 
Questionnaire  (PoQ).  (5  min)  (DISTRIBUTE  PoQ  TRANSCRIBING 
CLICK  COUNT  ONTO  TOP ) 

“( DISTRIBUTE  PoQ).  Please  take  a  moment  to  fill  out  this 
questionnaire  about  the  last  scenario  you  were  exposed  to.  There  is  no 
“penalty”  for  guessing  and  you  may  complete  the  questionnaire  in  any 
order  with  no  time  limit.  ”  ( Offer  screen  capture  for  MMC)” 

Subject  is  ushered  into  MCT  again.  Forth  and  last  scenario  is  initiated. 
Researcher  #1  will  initiate  the  next  specific  training  for  the  last  scenario. 
The  last  scenario  is  initiated  according  to  the  established  order  for  each 
participant.  (5  min)  (GIVE  SPECIFIC  TRAINING  AND  SET  UP 
SCENARIO  4) 

Forth/Last  subject  scenario  terminated  and  fill  in  post-experiment 
questionnaire.  Subject  briefed  to  wait  in  designated  holding  area  for  next 
scenario  preparation;  and  a  Post-  experiment  questionnaire  is 
administered.  See  Appendix  X  for  full  text  of  Post-Experiment 
Questionnaire  (PoQ).  (5  min)  ( DISTRIBUTE  PoQ ) 
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“  {DISTRIBUTE  PoQ ).  Please  take  a  moment  to  fill  out  this 
questionnaire  about  the  last  scenario  you  were  exposed  to.  ” 

T  +76  Subject  is  notified  of  experiment  termination.  (OFFER  COMMENT 

SHEET) 


***One  Subject  Evaluated  at~T  +  80*** 

T+  Simultaneously  Researcher  #2  resets  the  MCT  for  the  next  subject  (3 

mins)  then  will  go  administer  the  Pre-Experiment  Questionairre  (PreQ).  (5 
mins) 

(STEP  1,  Subject  #2)  Subject  is  ushered  into  controlled  scenario 
room/section  of  area.  Researcher  #1  reads  script  of  process  and  briefs 
subject  on  the  controls  and  familiarizes  them  with  system  that  will  remain 
the  same  throughout  all  three  scenarios.  This  will  constitute  the  training 
for  the  subject’s  interaction  with  system.  Any  questions  are  answered  to 
ensure  understanding  of  performance  for  subject.  All  scenarios  will  be 
briefed,  as  in  what  the  three  different  types  consist  of  and  infonn  the 
subject  that  the  order  of  presentation  will  be  random,  however,  operation 
of  the  system  will  be  the  same.  (10  mins) 
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Appendix  E:  Pre-Questionnaire 


Pre-Experiment  Questionnaire 

1 .  What  is  your  participant  number?  _ 

2.  What  is  your  gender?  _ 

3.  What  is  your  ATC  experience  (yrs/mos.)  _ 

4.  What  is  your  age?  _ 

5.  What  is  your  highest  level  of  education?  _ 

6.  Do  you  wear  contacts  or  glasses?  _ 

If  so,  are  you  wearing  them  now?  _ 

7.  Have  you  had  corrective  eye  surgery?  _ 

8.  Do  you  have  any  depth  perception  deficiencies?  YES  NO 

9.  Do  you  have  aviation  experience?  YES  NO 

If  YES,  what  type?  _ 

How  many  years  in  the  field?  _ 


10.  On  a  scale  of  1  to  5,  please  answer  whether  or  not  you  agree  with  the  following 
statements  (circle  one  of  the  numbers).  There  is  no  right  or  wrong  answer. 

Strongly  Strongly 

Disagree  Neutral  Agree 

A.  I  can  be  distracted  easily  while  working  ATC. 

1 

2 

3 

4 

5 

B.  I  have  trouble  multitasking  with  many  tasks. 

1 

2 

3 

4 

5 

C.  I  often  ask  pilots  to  repeat  themselves  when  stepped 

on.  1 

2 

3 

4 

5 

D.  I  think  that  new  technologies  can  improve  ATC. 

1 

2 

3 

4 

5 

E.  I  watch  movies  and/or  play  videogames  on  3-D  displays.  1 

2 

3 

4 

5 

F.  I  use  instant  messenger  text  correspondence  often. 

1 

2 

3 

4 

5 

G.  I  like  to  text  a  lot. 

1 

2 

3 

4 

5 

H.  I  consider  myself  a  fast  reader. 

1 

2 

3 

4 

5 
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I.  If  cost  was  not  an  issue,  I  would  prefer  3D  entertainment.  1  2  3  4  5 

J.  I  am  not  distracted  by  new  on-screen  text  while  working.  1  2  3  4  5 

K.  I  rely  heavily  on  computer  inputs  for  my  duties.  1  2  3  4  5 

L.  I  don’t  feel  confortable  listening  and  reading  at  the  same  time  1  2  3  4  5 

M.  What  percentage  of  your  total  day  is  on  a  computer?  0  25  50  75  100 

N.  I  have  experienced  dizziness  when  viewing  3-D  movies.  YES  NO 

O.  I  feel  that  ATC  is  an  important  job.  YES  NO 

P.  I  feel  3-D  displays  show  accurate  imagery  and  depth  YES  NO 


Thank-You  for  your  honest  responses  and  participation  in  this  study. 
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Appendix  F:  Training  Scenario 
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Appendix  G:  Training  Questionnaire 


Training  Questionnaire 

1 .  What  is  your  participant  number?  _ 

2.  Specifics:  (answer  as  you  remember) 

A.  Who  was  the  departure  aircraft? _ 

B.  Who  changed  their  type  of  approach  request? _ 

3.  While  monitoring  the  traffic  scene,  I  observed  the  following:  (circle  one  for  each 
choice) 

A.  The  highest  aircraft  in  the  scenario  was  (  ZOOM15  ,  STORMIO  or  EAGLE  15  ) 

B.  Which  altitude  did  WAGES54  requested?  (  FL180  FL230  or  130  ) 

4.  On  a  scale  of  1  to  5,  please  answer  whether  or  not  you  agree  with  the  following 
statements  (circle  one  of  the  numbers).  There  is  no  right  or  wrong  answer. 

No  Somewhat  Yes 

A.  I  am  comfortable  with  this  simulation  1  2  3  4  5  6  7 

B.  I  was  able  to  understand  the  pilots  clearly  1  2  3  4  5  6  7 

Thank-You  for  your  honest  responses  and  participation  in  this  study. 
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Appendix  H:  Post-Questionnaire  Control  Scenario 

Post  Questionnaire  (Be) 

1.  What  was  your  participant  ID  number? _ ( Researcher  use  only: _ ) 

2.  Specifics:  (answer  as  you  remember.  Guess  rather  than  leaving  blank) 

A.  What  type  of  emergency  was  EAGLE15? _ 

B.  What  altitude  did  OPEC26  H  initially  climb  to? _ 

C.  One  aircraft  requested  to  cancel  IFR.  Which  one? _ 

D.  What  altitude  was  JOKER22  requesting? _ 

E.  One  aircraft  requested  direct  point  bravo.  Which  one? _ 

F.  Who  changed  its  type  of  approach  from  vectors  for  ILS  to  direct  initial? _ 


3.  While  performing  the  air  traffic  control  simulation:  (please  circle  one) 

No  Somewhat  Yes 


A.  I  felt  distracted  during  the  scenario. 

B.  I  felt  the  task  was  over  simplified. 

C.  I  found  it  difficult  to  maintain  concentration. 

D.  The  task  seemed  too  difficult. 

E.  The  scenario  seemed  realistic. 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


6  7 
6  7 
6  7 
6  7 
6  7 


4.  In  general,  each  time  the  aircraft  stepped  on  each  other  (Circle  all  that  apply.  You  can  choose 
more  than  1  option): 

a.  I  didn't  understand  either  aircraft 

b.  I  understood  the  call  sign  (ACID)  of  at  least  one  aircraft 

c.  I  could  understand  the  request  of  at  least  one  of  them 

d.  Completely  understood  one  aircraft 

e.  I  copied  all  and  was  able  to  answer/make  a  decision 
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Appendix  I:  Post-Questionnaire  Experimental  Scenario 

Post  Questionnaire  (Be) 

1.  What  was  your  participant  ID  number? _ ( Researcher  use  only: _ ) 

2.  Specifics:  (answer  as  you  remember.  Guess  rather  than  leaving  blank) 

A.  What  type  of  emergency  was  STORM  28?  _ 

B.  What  altitude  did  N172FA  descend  to? _ 

C.  One  aircraft  requested  transition  to  delta  surface  area.  Which  one? _ 

D.  What  altitude  was  ORCA  27  H  requesting? _ 

E.  One  aircraft  requested  vectors  to  straight  in  touch-n-go  option  tower.  Which  one? 

F.  Who  changed  its  type  landing  from  full  stop  to  option  tower? _ 


3.  While  performing  the  air  traffic  control  simulation:  (please  circle  one) 

No  Somewhat  Yes 


A.  I  felt  distracted  during  the  scenario.  1 

B.  I  felt  the  task  was  over  simplified.  1 

C.  I  felt  confident  relying  on  the  on-screen  text  cues.  1 

D.  I  found  it  difficult  to  maintain  concentration.  1 

E.  The  task  seemed  too  difficult.  1 

F.  The  scenario  seemed  realistic.  1 

G.  I  think  the  additional  text  cueing  was  useful.  1 

H.  I  would  like  to  work  more  with  additional  texts.  1 

I.  I  found  it  difficult  to  use  the  text  cueing.  1 

J.  I  used  the  text  cueing  to  de-conflict  the  audio.  1 

K.  Even  if  I  can  read  it  in  real  time,  I  prefer  to  hear  it  from  pilots  1 


2  3  4  5  6 
2  3  4  5  6 
2  3  4  5  6 
2  3  4  5  6 
2  3  4  5  6 
2  3  4  5  6 
2  3  4  5  6 
2  3  4  5  6 
2  3  4  5  6 
2  3  4  5  6 
2  3  4  5  6 


7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 
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4.  In  general,  each  time  the  aircraft  stepped  on  each  other  (Circle  all  that  apply.  You  can  choose 
more  than  1  option): 

f.  I  referred  to  the  text  box 

g.  I  didn't  understand  either  aircraft 

h.  I  understood  the  call  sign  (ACID)  of  at  least  one  aircraft 

i.  I  could  understand  the  request  of  at  least  one  of  them 

j.  Completely  understood  one  aircraft 

k.  I  copied  all  and  was  able  to  answer/make  a  decision 

5.  I  find  the  text  to  be  useful  in  this  capacity  (please  circle  one)  Yes  No 

6.  I  see  potential  for  the  technology  in  this  or  other  capacities,  (please  circle  one)  Yes  No. 

If  yes,  please  briefly  explain  your  opinion: 


7.  I  have  ideas  to  improve  on  this  scenario/application  and  I  am  willing  to  submit  additional  inputs 

to  the  researcher  to  improve  this  experiment,  (please  circle  one)  Yes  No 

8.  Please  indicate  whether  you  felt  voice  conflicts  were  easier  to  understand  with  the  text  cues 
and  why: 


9.  Often  controllers  choose  to  employ  strategies  in  an  attempt  to  make  decisions  and  improve 
performance.  Did  you  find  the  additional  text  cueing  to  be  conducive  or  helpful  in  your  decision 
making? 

Yes  No  If  yes,  please  briefly  describe  how  it  influenced  your  performance,  if  at  all: 
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pilots’  transmissions. 
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